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ABSTRACT 


This thesis presents the design, implementation and evaluation of 
two abstracted programming and communication interfaces for devel- 
oping distributed programs on a network of Transputers. One inter- 
face uses a shared memory model for interprocess communication and 
synchronization. The other interface uses a message passing model 
for communication and synchronization. The programming interfaces 
allow development of distributed programs that are independent of 
the physical configuration of a network. This thesis also presents an 
evaluation of Transputer performance with a particular emphasis on 


the interaction of computation and inter-Transputer communication. 
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I. INTRODUCTION 


A. BACKGROUND 

The Aegis Modeling Project in the Computer Science Department 
at the Naval Postgraduate School is engaged in researching advanced 
computer architectures for potential future application aboard naval 
ships. Currently, this research is centered on the application and 
evaluation of distributed computing architectures. A distributed 
architecture is particularly suited for shipboard applications. Ship- 
board locations at which processing capabilities are required are 
physically distributed, yet processors at all locations must cooperate to 
monitor and control a ship's sensors and systems. Such an architec- 
ture can also provide the necessary characteristics of high perfor- 
mance, fault tolerance, and extensibility. 

One emphasis of the current research has been to investigate sys- 
tems that are composed of relatively low cost, “off-the-shelf” compo- 
nents. The single chip microprocessor is such a component. A range 
of microprocessor-based distributed multicomputer systems are, 
therefore, being evaluated. One such system already developed and 
evaluated uses clusters of microprocessor-based single-board comput- 
ers interconnected by a hierarchical bus structure [Ga86]. A dis- 
tributed system architecture now being evaluated is based on the 


single chip microprocessor known as the Transputer. 


B THE TRANSPUTER 

The Transputer is a single chip microprocessor that has been 
specifically designed to function as a computing element in a dis- 
tributed multicomputer system. The name “Transputer” is an amal- 
gam of the words transistor and computer. As the transistor was a 
building block for large and varied electronic circuits, the Transputer 
is intended by the manufacturer to be an analogous building block for 
distributed computing systems. 

To facilitate the use of the Transputer as an element in a dis- 
tributed system, the Transputer implements the concept of Commu- 
nicating Sequential Processes [Ho79]. Communicating Sequential 
Processes is a paradigm which defines and describes the interaction of 
programs that execute in parallel (as is the case in a distributed 


system). 


C ABSTRACT PROGRAMMING INTERFACES 

Program development for a network of Transputers is currently 
closely tied to the particular physical configuration of a given network. 
The physical configuration of a network must be considered early and 
throughout the software design process. In certain cases, the configu- 
ration can actually dictate aspects of software design. This thesis 
investigates isolating the software designer from this physical configu- 


ration through the use of an abstract programming interface. 


D. AVAILABLE HARDWARE AND SOFTWARE 

The Transputer hardware available to the Aegis Modeling Project 
is varied. The hardware includes Transputer interface cards for per- 
sonal computers, Transputer-based serial interface and color graphics 
interface cards, and cards with multiple Transputers. Aspects of this 
hardware that pertain to portions of this thesis are discused where 
applicable. A complete description of the hardware is available 
elsewhere [In86]. 

Programs included in this thesis were developed using the Trans- 
puter Development System(TDS) [In87a] and the OCCAM program- 
ming language [PoMa87]. OCCAM is a high-level block-structured lan- 
guage which includes constructs based on CSP to support program- 
ming in a distributed environment. An assembler and Pascal and C 


compilers for the Transputer are also available. 


E. THESIS ORGANIZATION 

Chapter II describes the Transputer and the basic concept of 
Communicating Sequential Processes. 

This thesis investigates the basic performance characteristics of a 
Transputer as an element in a network of Transputers. Chapter III 
describes testing that was performed and documents the results of 
this testing. 

Chapter IV and Chapter V describe and evaluate two different 
prototype abstract programming interfaces developed for a network of 
Transputers. One interface is based on a virtual globally shared mem- 


ory. The other is based on message passing. 


Chapter VI presents the conclusions reached as a result of devel- 
oping, using, and evaluating the programming interfaces. Recommen- 


dations for further research are also provided in Chapter VI. 


Il. THE TRANSPUTER 


A OVERVIEW 

Central to the to the Transputer is the concept of Communicating 
Sequential Processes (CSP) [Ho79]. The Transputer is, in fact, a hard- 
ware implementation of this concept. The programming language 
OCCAM, which is the primary language used for programming the 
Transputer, is based on this concept. A summary of CSP is presented 
in this chapter. 

To effectively evaluate and use the Transputer requires an under- 
standing of how the Transputer implements the concept of CSP. In 
addition, in many cases this thesis refers to Transputer architectural 
features and details. This chapter, therefore, also presents a brief 


description of Transputer architecture. 


B COMMUNICATING SEQUENTIAL PROCESSES 

Communicating Sequential Processes (CSP) is one model for con- 
current or parallel programming. In CSP, a program is composed of 
processes. A process consists of a list of commands or instructions 
that are to be executed in Sequence. The different processes within a 
program are combined and specified to be executed in Sequence or in 
parallel or in some sequential/parallel combination. The data spaces 
for any processes executed in parallel are constrained to be disjoint. 

This requirement for parallel processes to have disjoint data 


Spaces precludes using shared memory to communicate between the 


processes. To provide for necessary process-to-process communica- 
tion, CSP instead utilizes message passing. Messages are passed 
between any pair of parallel processes via synchronous, unbuffered, 
point-to-point communications channels connected between the 
processes. These communication channels also provide the means for 
synchronizing processes. To communicate between two processes, 
one process must include an instruction for performing an output to 
the other process and the other process must include a corresponding 
input instruction. Both processes must be at that point in their 
instruction execution where the communication is specified to occur 
(If one process reaches this point first, it waits for the other process 
to reach its point of communication). The communication is ™aiea 
performed. Since the communication only occurs when both process 
are at their points of communication, the processes are synchronized 
at these points. In addition, CSP includes constructs for program 
control and sequencing and for conditional selection between multiple 
communications. 

Figure 2.1 depicts a set of processes (represented as circles) 
interconnected by point-to-point communications links (represented 
as directed lines). This set of processes would operate independently 
and in parallel on a continuous stream of input values to produce a 


continuous stream of output values. 


Processes to Calculate Unity Based on the Formula 


gee 2 
SIN X + COS X 





Figure 2.1. Process Representation Exampl 





Groups of processes may be logically aggregated to form larger, 
more abstracted process constructs. Figure 2.2 shows one possible 


abstraction of a subset of the processes shown in Figure 2.1. 


Processes to Calculate Unity Based on the Formula 


mae a 
SIn. X + COS X 





Figure 2.2. Process Abstraction Example 





C TRANSPUTER ARCHITECTURE 

A block diagram of a typical Transputer is shown in Figure 2.3. 
This figure depicts the major architectural components a Transputer. 
The following sections give a brief description of each of these com- 
ponents. In particular, these sections point out some architectural 
aspects of the Transputer that can influence the performance of pro- 
grams written for the Transputer. 

Several versions of the Transputer are currently available. This 
thesis considers only Transputer types T414 and T800. For this rea- 
son, the following sections describe the features of these Transputer 
types. A complete description of all currently available Transputers 
can be found elsewhere [In87b]. 

1. Processor 

In general, the processor consists of a 32-bit integer arith- 
metic unit and a set of 32-bit registers. Figure 2.4 shows a block dia- 
gram of the processor. The I or instruction register points to the next 
instruction to be executed in a process. The W or workspace register 
is a pointer to the beginning of an area in memory that is the data 
space for a process. Together, these two registers can be thought of 
aS representing the instructions and the data for a single process. The 
A, B, and C registers form a push-down evaluation stack. In general, 
all operations are performed on or using the values in these registers. 
For example, the load and store operations load and store the value of 
the A register. The add operation adds the values of the A and B 


registers leaving the result in the A register. 
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Figure 2.4. Processor Block Diagram 


a Instruction Set 

Each byte in a program can be viewed as having a four-bit 
high-order half and a four-bit low-order half. The low-order half of an 
instruction byte contains a data value. The high-order half contains a 
function code. To execute such an instruction, the data value half of 
the instruction byte is loaded into the operand register and the func- 
tion encoded in the other half of the instruction byte is performed. At 
this point, it would appear that this instruction format limits operand 
values to the range of 0 to 15 and that only 16 different functions 
could be performed. A means is provided, however, for representing 
larger data values and for encoding a greater number of functions. The 
O or operand register is used to form operands and multi-byte 


instructions from a Sequence bytes. For data values represented by 


We 


more than four bits, special function codes cause individual four bit 
“pieces” of the data value to be extracted from a series of instruction 
bytes and accumulated in the operand register. The last function code 
in this series of instruction bytes will actually operate on the accumu- 
lated data value. Figure 2.5 shows an example of this operand forma- 


tion process. 
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Figure 2.5. Example of Operand Register Operation 
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To encode functions with value representations greater 
than four bits, the value representing such a function is loaded into the 
operand register in the same manner as if it were data. A special 
function code then causes the value in the operand register to be exe- 
cuted as an instruction. Using this instruction formatting scheme, the 
instruction set has been optimized so that the most frequently used 
operations are encoded using only a single byte. Measurements show 
that about 70% of the instructions actually executed in a typical pro- 
gram are, in fact, encoded using only a single byte [In87c]. Having 
most instructions represented as single bytes not only reduces the 
memory requirements for program code but tends to improve pro- 
gram performance. This is because, since fewer bytes are fetched per 
instruction executed, fewer memory accesses will be required to exe- 
cute the program. 

This method of encoding instructions has other effects 
on processor performance. Each byte of a multi-byte instruction takes 
one processor clock cycle to load into the operand register for assem- 
bly of the instruction or its operand. Because of this, instructions with 
a greater byte length take more time to assemble before they can be 
executed. In most cases, the length of an instruction is fixed. How- 
ever, in some cases, the length of an instruction is dependent on the 
value or location of the instruction’s operand. Since these factors can 
be controlled by the programmer, it is useful to be aware of these 
types of instructions. For example, data values located in the first 16 


locations above the base of the processor workspace require only one 


instruction byte to be accessed. Data values located elsewhere require 
additional instruction bytes, which increases the time required to 
access these data values. Because of this, frequently accessed data val- 
ues should be located closest to the base of the workspace. Also, 
loading a constant takes one instruction byte for each four bits of the 
constant’s length. This means that loading a 32-bit constant takes 
eight processor clock cycles. If such a constant is to be used 
repeatedly, it is often more efficient to name and store the constant as 
a data value and use that data value instead. 
b. Concurrency 

A single Transputer directly supports running concur- 
rent processes. These processes may be either of two priority levels: 
high or low. To facilitate implementation of concurrency and ‘prioriti- 
zation, the Transputer has a micro-coded process scheduler. The 
operations performed by this scheduler can be examined based on 
whether a process is active or inactive. An inactive process is one that 
is waiting on communication or on a programmed time delay 
(operations for inactive processes are discussed later in this chapter). 
An active process is one that is not waiting and is ready to execute. 

To manage the active processes, the process scheduler 
maintains a separate linked list of active processes for each priority 
level. Instructions are included in the instruction set for starting new 
processes by adding the process to an active process list. Processes 
are selected for execution from these active process lists based on the 


following generalized rules: 
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e« High-priority processes are always executed in preference to low- 
priority processes. If a low-priority process is executing when a 
high-priority process is added to the high-priority active process 
list, the low-priority process is preempted and the high-priority 
process is executed. A high-priority process is executed until it 
completes or until it must wait for communication or for a pro- 
grammed time delay. 

« When no high-priority processes are available for execution, a low- 
priority process may be executed. A low-priority processes is 
executed until it must wait for communication or forsagpee 
grammed time delay. In addition, to ensure that one low-priority 
process does not monopolize the processor, a low-priority pro- 
cess that has been executing for more than about one millisecond 
is suspended. The suspended process is placed at the end of the 
low-priority active process list and the process at the beginning of 
the low-priority active process list is then executed. 


2. Fioating Point Unit 
One version of the Transputer includes hardware for per- 
forming floating point arithmetic operations. Internally, the floating 
point unit includes a three-register floating-point evaluation stack that 
operates in the same manner as the “normal” or integer processor’s 
evaluation stack. The floating-point unit operates in parallel with the 
other components of the Transputer. Floating-point operations may, 
therefore, be performed in the floating-point unit at the same time 
that integer calculations are being performed in the integer processor. 
Currently, only the Transputer model T8000 includes this floating- 
point unit. 
3. Links 
The hardware links provide the means for implementing CSP 
communication channels between processes executing on different 
Transputers. A link from one Transputer is connected to a link on 


another Transputer to provide a bidirectional pair of communications 
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channels. Each hardware link can perform simultaneous, independent 
input and output communication. Instructions are included in the 
Transputer’s instruction set for performing input and output opera- 
tions using the links. A block diagram of a communications link is 
shown in Figure 2.6. Each communications link consists of an inde- 
pendent direct memory access controller and serial communication 


logic. 
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Figure 2.6. Hardware Communication Link Logical Block Diagram 


To perform a communication via a hardware link, the com- 
munication link address and the size and location of a message are 
specified, then the communications instruction is executed. This ini- 
tializes the direct memory access controller with the size and location 


of the message to be communicated. The process executing the 


Les 


communication instruction is suspended and a pointer for Tater 
resuming the process is saved (another ready process from an active 
process list may then be executed). When both the sending and 
receiving links have been initialized in this manner, the message 
communication is accomplished. When the communication is com- 
plete, the process which was suspended for communication is added 
to the end of the appropriate high- or low-priority active process list 
to wait its turn for execution. 

Messages are transmitted by the links one byte at a time in a 
bit-serial format. After a receiver has recognized the reception of a 
byte and is capable of receiving another byte, the receiver transmits an 
acknowledge message. The transmitter will await reception of the 
acknowledge message before transmitting the next message byte. 
Since the link hardware performs no error checking on messages, the 
purpose of the acknowledge message is solely to control the flow of 
message bytes between the links. Figure 2.7 shows a formatted mes- 


sage byte and an acknowledge message. 
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Figure 2.7. Serial Communication Link Protocol 
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A method for communicating between processes executing 
on the same Transputer is also provided. To perform such a commu- 
nication, a memory address and the size and location of a message are 
specified, then the communications instruction is executed (note that 
this is the same Sequence required to initiate communication via a 
link). When both the sending and receiving processes are ready to 
communicate, the communication is accomplished by performing a 
memory-to-memory transfer of the message data. 

4. Memo 

The Transputer can address four gigabytes of memory. This 
memory space is divided into two non-overlapping segments: an 
internal or on-chip segment of memory and an external or off-chip 
segment of memory. Although these two segments of memory are 
logically the same, the physical characteristics of the two segments 
are quite different. 

The on-chip memory consists of up to four kilobytes of static 
random access memory (depending on the type of Transputer being 
considered). Within the address space, the on-chip memory occupies 
the lowest block of addresses. The on-chip memory can be accessed 
for read or write via the internal processor bus in one processor clock 
Evcle. 

The off-chip memory forms the balance of the address space. 
The off-chip memory is accessed via an external memory interface. 
This external memory interface provides access to the external mem- 


ory by multiplexing a 32-bit address and 32 bits of data onto a single 
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32-bit external bus. The timing for this multiplexing slows access to 
the external bus to a minimum access time of three processor clock 
cycles. The actual number of cycles required to access external mem- 
ory is also dependent on the requirements of the external memory 
devices. For the Transputer systems available in our laboratory, the 
external memory access times range from three to five processor 
clock cycles. This difference in access times between the “fast” on- 
chip and “slow” off-chip memory can affect program performance. 
Frequently accessed variables or code segments should, therefore, be 
preferentially located in the “fast” on-chip memory. 

Additionally, the external memory interface includes control 
logic for refreshing external dynamic random access memory devices. 
Control lines and signals are also provided to facilitate peripheral 
device direct memory access to the external segment of memory. 

5. Timers 

The Transputer provides two hardware timers. Each of these 
timers can be viewed as free-running 32-bit binary counters. One of 
the timers is accessible to high-priority processes; the other timer is 
accessible to low-priority processes. The high-priority timer incre- 
ments at intervals of one usecond, for a total cycle time of about 72 
minutes. The low-priority timer increments at intervals of 64 
useconds, for a total cycle time of about 76 hours. 

Instructions are provided for initializing the value of these 
timers and for reading the current value of a timer. An instruction is 


also provided for suspending execution of a process until a specified 


18 


timer value is reached. To implement this instruction, the Transputer 
maintains a linked list of suspended processes waiting on timer values. 
Separate lists are maintained for the high- and low-priority timers. 
These lists are ordered by the specified “wait-until” time value. The 
first “wait-until” time value in each list is loaded into a dedicated reg- 
ister and, using hardware, is compared with the current value of the 
high- or low-priority timer. When the “wait-until” timer value is 
reached, the suspended process is removed from the timer list and is 
added to the end of the appropriate active process list to wait its turn 
for execution. The next timer list entry is then loaded for comparison. 
6. External Event Input 

The external event input on the Transputer is similar to an 
external interrupt input. To the Transputer, this external event input 
appears as a communications channel which is capable of transmitting 
a signal to a user’s program. A user’s program requesting input from 
the external event channel will be suspended if the external event 
input is not being asserted. Then, when the external event input is 
asserted, the process will be added to the end of an active process list 
to wait its turn for execution. Either a high- or a low-priority process 


may request input from the external event channel. 
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III. TRANSPUTER PERFORMANCE 


A OVERVIEW 

As has been described in the previous chapter, the Transputer is a 
complex microprocessor. Because of this complexity, the perfor- 
mance of even a single Transputer can be affected by many factors. 
Such factors might include whether or not the program and/or 
associated data is in on-chip or off-chip memory or if there is internal 
bus contention resulting from external communications link direct 
memory access. When considering a network of Transputers, the fac- 
tors that can affect overall network performance are multiplied con- 
siderably. 

Developing efficient Transputer-based systems in the face of this 
complexity requires a firm understanding of Transputer performance 
and the manner in which different factors influence that performance. 
Evaluating Transputer-based systems requires accurate methods for 
measuring the performance of individual and networked Transputers. 

To begin to understand Transputer performance characteristics 
and to gain experience in measuring individual and networked Trans- 
puter performance, a series of timing and performance studies was 


conducted. 


B PRIOR RESEARCH 
INMOS provides basic performance specifications for the 


Transputer [In87b and In87d]. The basic specifications list the 
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performance for individual machine-level and high-level language 
operations. The listed specifications consider the effects of some of 
the factors that can potentially affect performance. While it is 
expected that the manufacturer’s performance specifications are 
accurate, it is necessary to independently confirm this. 

Prior theses [Va87 and Ha87] document some detailed tests and 
analyses of Transputer performance. These theses point out, however, 
that some aspects of their performance test results do not appear to 
be consistent or cannot adequately be explained. They suggest that 
further research be performed in this area to resolve the identified 


problems and to extend the scope of performance testing. 


C TEST METHODOLOGY 

This chapter documents the suggested further timing research 
and documents the results of an initial set of timing tests on the 
recently released T800 20 MHz Transputer. Two major categories of 
testing are addressed. The first category is testing to confirm the 
basic manufacturer’s performance specifications for the Transputer. 
The second category is testing to determine the interaction that exists 
between the operation of the central processing unit and communica- 
tion link direct memory access activity. 

1. Con ation 

To accomplish both types of testing, a single test configura- 

tion was developed. The test configuration consists of a central 
“target” Transputer and four “satellite” Transputers, each attached to 


the target Transputer by a communications link. In addition, there 
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are associated Transputers which perform the functions of control and 
of data routing and recording. A logical diagram of this test configura- 
tion is presented in Figure 3.1. A detailed diagram of the test set-up is 


presented in Appendix A. 


{ Satellite 
Processor 


Satellite 
Processor 





Although based on and quite similar to a previously used test 
configuration [Ha87], there is a subtle but significant difference 
between the two test configurations. In the prior test configuration, 
one of the satellites was the Transputer installed in the host develop- 
ment system. User programs executed on the Transputer in the host 
development system are run from within the Transputer Development 


System (DS) sesmellas 


2) 


After some initial experimentation, it became apparent that, 
in some way, this shell affected the timing of programs run from 
within the shell. The timing of programs running on Transputers 
external to the shell but depending on communications to or from a 
program run from within the shell also appeared to be affected. It is 
postulated that some shell processes are active while the user pro- 
gram is executing and, to an extent, interfere with the user program. 
References to such processes can be found in [In87a]. 

Although it may be of interest to research this aspect of TDS 
in the future, it is sufficient for the purposes of the current timing 
tests to simply ensure that all timing measurements are performed 
external to and independent of the host development system 
Transputer. 

2. Software 

In general, the target Transputer performed some calculation 
in a loop while the satellite Transputers placed different link input 
and output communications loads on the target Transputer. The test 
software monitored the time required to perform link communica- 
tions and the number of calculation loops that could be performed 
during those communications. Appendix A provides a listing of the 
programs used to measure and record processor performance. 

3. Conditions 

Because so many factors can interact and affect Transputer 

performance, it was necessary to set and hold some test conditions 


fixed so that the effects of variations in parameters of interest could be 
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properly interpreted. Constant for the tests documented by (tame 


thesis were: 


T414 Transputer processor speed was 15 MHz. 
T800 Transputer processor speed was 20 MHz. 


Communications link speed was fixed at 20 megabits per second 
for both Transputer types. 


Program code and scalar variable values in the calculation loop 
were located in the “fast” on-chip memory for both Transputer 


types. 


Scalar variables in the calculation loop were in “local” scope {i.e., 
within the first 16 bytes of the bottom of the workspace and 
accessible by single byte load and store instructions). 


Memory-to-memory transfer blocks were located in “slow” off- 
chip memory for both Transputer types. 


Communications data blocks were fixed at 100,000 bytes. 
Because this block size is much greater than the size of the on- 
chip memory, the block was located in “slow” off-chip memory 
for both Transputer types. 


Communications processes were run at high priority and the cal- 
culation loop was run at low priority. 


Time measurements were taken in the each Transputer using the 
high-priority one-usecond resolution timer. 


The effects of varying certain parameters of interest were 


investigated. The following list identifies the parameters of interest 


and the manner in which they were varied. 


e The characteristics of the target Transputer calculation loop were 


varied by placing different types and numbers of operations within 
the loop. The operations used within the loop were no operation, 
assignment, addition, subtraction, multiplication, division, and 
100- and 1000-byte memory-to-memory transfers. The number 
of individual operations of a type in a loop ranged from O to 4 
operations per loop. 
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¢ Communications conditions were varied by changing the number 
of links that were active at one time. Additionally, the communi- 
cations conditions were varied by changing the size of a commu- 
nications packet. The number of active communications links 
ranged from O to 4 input links active and from 0 to 4 output links 
active. The individual data packet size was varied in 16 steps 
from one byte per packet to 100,000 bytes per packet with an 
overall constant data block size of 100,000 bytes. 


4. Data Management 

From the number of possible combinations of test conditions, 
it was clear quite early that these timing tests would generate a large 
amount of test data. To more effectively handle this test data, it was 
decided to record the data in some database management system. 
Using a database management system provided for ease of access to 
selected aspects of the data. Additionally, since the potential exists 
for performing further timing tests, such a system will facilitate the 
incorporation and aggregation of any future timing data. 

Since the Ingres relational database management system was 
readily available on the departmental mini-computer, this system was 
selected for use. To facilitate loading of the timing database, condi- 
tions for a particular timing test and the test results were directed to 
a disk file on the host development system, then electronically trans- 
ferred to the departmental mini-computer for loading into the Ingres 
timing database. Appendix B describes the Ingres database created for 
the timing data and gives some examples of accessing timing informa- 


tion from the database. 
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D. TEST RESULTS 
Test runs covering the range of conditions described previously 
were completed and the resulting data was loaded into the Ingres 
timing test database. Data was extracted from this database to “view” 
several aspects of Transputer performance. These aspects of perfor- 
mance are: 
1. Isolated Processor Performance 

The first view of the timing results is to compare measured 
instruction execution times with the manufacturer’s specified instruc- 
tion times for a selected subset of Transputer instructions. This is 
intended as a confirmation of the manufacturer’s stated execution 
times. Additionally, if the timing results are consistent with the man- 
ufacturer’s specifications, it will tend to validate the timing test 
methodology that is being used. 

Determining the manufacturer’s specified execution time for 
a selected loop operation requires that the operation be examined at 
the machine-language level. Using a disassembler [Br87] to facilitate 
examination of the timing program object code, each of the selected 
Operations was decomposed into its machine-level components. The 
manufacturer-specified number of processor clock cycles for each of 
these components of an operation was summed, then multiplied by 
the period of the processor clock. The result of this calculation is the 
expected execution time for a particular loop operation executed in 
on-chip memory. For example, the expected execution time for the 


multiply operation is determined as follows: 


26 


Multiply Operation: 
a:=b*c 


Machine-Language Equivalent Multiply Operation: 


Id] b (2 cycles) 
Id] C (2 cycles) 
pfix; mult (1 + 38 cycles) 
stl a (levcle) 


Execution Time Calculation (T800 @ 20 MHz): 


44 cycles x cer an = 2.200 usec 

Note that execution of instructions in “slow” off-chip memory 
affects the calculation of expected execution times. To determine the 
expected time for such an off-chip operation, the same basic method 
described above is used, except that a separate accounting of on-chip 
and off-chip cycle counts is maintained. The off-chip cycle count is 
then multiplied by a hardware-dependent scale factor. This scale fac- 
tor is the number of processor cycles required to make a single access 
to the off-chip memory. In the case of the hardware used for these 
timing tests, this scale factor is 4 for the T414 Transputer and 5 for 
the T800 Transputer. 

In addition, in the particular hardware configuration used for 
these timing tests, off-chip memory consisted of dynamic random 
accesS memory devices. Such devices must periodically be 
“refreshed” to maintain their data. Although this refresh operation is 


relatively fast and is handled automatically by the Transputer 
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hardware, access to the off-chip memory is restricted during a short 
period of time. In the worst case, this increases the average time 
required to access off-chip memory by a factor of 1.0237 for the T414 
Transputer and 1.0213 for the T800 Transputer [In87b]. The worst- 
case overall time for an off-chip operation is, then, the sum of the on- 
chip time and the scaled and “refresh delayed” off-chip times 
example calculation for execution of an off-chip memory-to-memory 


transfer follows: 


Memory-to-Memory Transfer Operation: 
[array FROM O FOR 1000] := [array FROM O FOR 1000] 


Machine Language Equivalent Operation: 


pfix; pfix; ldc #$800 (1 + 1 + 1 cycles) 

pfix; mint (1 + 1 cycles) 

wsub (2 cycles) 

stl temp (1 cycle) 

Dice O ix elec #3800 (1 + 1 + 1 cycles) 

pfix; mint (1 + 1 cycles) 

wsub , [2 -cveles) 

ldl temp (2 cycles) 

pfix; pfix; ldc #$3E8 (1 + 1 + 1 cycles) 

pfix; move ; (1 + 8 cycles maximum 


500 off-chip cycles) 


Execution Time Calculation (T414 15 MHz): 


.067 
29 reveles = ae = |. IGS see 
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500 cycles x oes x (4 x 1.0237) off-chip = 136.493 usec 


1.933 psec + 136.493 usec = 138.426 usec maximum 


Measured execution times for different operations were 


derived from the timing data as follows: 


Consider only timing data where no external communication links 
were active. 


Calculate the average time per loop for a test set with no opera- 
tions in the loop. (The average loop time is simply the time of 
measurement divided by the number of loops executed during 
that time.) 


Extract the average time per loop for a test set with 1, 2, 3, and 4 
instances of a selected operation in the loop. 


Subtract the no-operation loop time from each of the selected 
operation loop times. These loop times have now been “adjusted” 
to remove any loop overhead time. 


The incremental increase in adjusted loop times for the selected 
operation loops is the expected time required to perform a single 
operation. 


Tables 3.1 and 3.2 list the expected and measured execution 


times for the selected loop operations. These results show that, 


except for the divide operation, the measured results are consistent 


with the expected results. The expected time for the divide operation 


is based on the operation taking 39 processor clock cycles [In87d]. 


The measured results indicate that the divide operation takes 38 pro- 


cessor clock cycles. Subsequent to performing the timing tests, it was 


confirmed with the manufacturer that the divide operation, in fact, 


takes 38 clock cycles as measured in the timing tests [Pe88]. 
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TABEE Sa 


T414 EXPECTED AND MEASURED 
INSTRUCTION EXECUTION TIMES 


- Operation [Operations] Test | Loops Loop | Measured | Calculated 
per Duration | During Time Op Time | Op Time 
- Loop | __(sec) Test sec/Loop)| _(lsec) (sec) 
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TABLE 3.2 


T800 EXPECTED AND MEASURED 
INSTRUCTION EXECUTION TIMES 


Operation [Operations] Test Loops Loop | Measured | Calculated 
per Duration | During Time Op Time | Op Time 
_ Loop | __(usec) Test _|(usec/Loop)} _(LUsec) (usec) | 


NullLoop_[_____ 0] 1000009] 285381] 3.50 
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2. Isolated Communications Link Performance 
The Transputer manufacturer also provides specifications for 


communications link performance [In87b and In87c]. The measured 
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performance of a communications link can also be calculated from 
timing test data. Measured communications link performance is cal- 
culated by dividing the size of the largest single data block communi- 
cated over a single link by the time required for the communication. 
The largest block size was selected to minimize the affects of any 
communications set-up overhead that might exist. For the timing 
tests performed, the largest block size was 100,000 bytes. The speci- 
fied and measured communications data rates are compared in Table 
3.3. This table shows that the specified and measured data rates are 
consistent. 


TABLE 3.3 


ISOLATED COMMUNICATION LINK PERFORMANCE 
Processor | “Communication [ Specified Rate 
a TDes _ Mode __(Kbytes/sec) _ _(Kbytes/sec) 


[Input | 
T414 Output 
Input/Output 





Of particular interest here is the bidirectional data rate for 
the T800 Transputer. For the T414 Transputer, the bidirectional data 
rate is approximately 2 times the unidirectional data rate. For the 
T800, however, the bidirectional data rate is only about 1.35 times the 
unidirectional data rate. Since in the worst case of the external off- 


chip memory the processor bus data rate is at least 20 megabytes per 
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second, accessing data from memory should not limit link perfor- 
mance. Taking a closer look at the link communications protocol 
provides the key to understanding this problem. 

In unidirectional communication for the T800, the acknowl- 
edge packet from the receiving Transputer is returned to the trans- 
mitting Transputer before the transmitter completes its transmission. 
Because of this, the transmitter is able to transmit bytes “head-to-tail” 
without any intervening delays. This results in a maximum theoretical 


unidirectional data rate of 


20 Mbits 1 byte _ 1.82 Mbytes 
sec * 11 bits transmitted — sec 


which is consistent with the actual unidirectional link data rate. In 
bidirectional communications, however, each Transputer must 
“sandwich” acknowledge packets between data packets. This 
increases the total number of bits transmitted by a Transputer per 
data byte from 11 to 13 and results in a maximum theoretical bidirec- 


tional data rate of 


20 Mbits 1 byte one 3.08 Mbytes 
sec * 13 bits transmitted * ~ = sec 


This is less than double the unidirectional rate. Additionally, 
because a receiving Transputer is also transmitting its own data, it may 
not immediately be able to return an acknowledge packet for data 
received. The transmitter may, therefore, be delayed in transmitting 


its next data byte, further reducing the data rate. However, since the 
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internal timing threshold requirements of the link hardware are not 
known, it is not possible to exactly quantify this effect. 

Because of this bidirectional link communications limitation, 
two separate unidirectional communications links between T80O 
Transputers will provide about a 1.1 megabyte greater bidirectional 
communications throughput than will a single communications link 
operated bidirectionally. 

Since the T414 unidirectional communication rate is signifi- 
cantly less than the maximum possible rate for a 20 Mbit/second data 
link, space already exists between successive transmitted packets to 
“insert” an acknowledge packet for a received data packet. Because of 
this, the T414 does not show the prominent bidirectional 
communications protocol limitation shown for the T800. 

3. Communication Link Interaction 

Prior research [Va87] has shown that, for the T414 Trans- 
puter, there is minimal interaction between links when multiple links 
are operated simultaneously. It was desired to determine whether 
this is also a characteristic of the T800 Transputer with its greater 
communication link data rates. Figures 3.2 and 3.3 show communica- 
tions link performance for both the T414 and the T800 under condi- 
tions when multiple links are active. Except for the bidirectional 
protocol limitation previously discussed, these figures show that there 
is minimal interaction when multiple links are being operated. Note 
that in Figure 3.2, the number of links active includes both unidirec- 


tionally and bidirectionally active links. Because of this, each number 
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of active links may include more than one data point. For example, 
two links active on Figure 3.2 include the data points for both the case 
when two unidirectional links are active and the case when one bidi- 


rectional link is active. 
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Figure 3.2. T414 Multiple Link Effects on Communications Rate 





go Unidirectional 
@ Bidirectional 


(bytes/second/link) 


a 
om 
bs 
5) 
ey) 
v) 
= 
2 
ras) 
Q 
= 
= 
= 
= 
fe) 
Se, 


5 
Active Links 





Figure 3.3. T800 Multiple Link Effects on Communications Rate 
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Additionally, for the T414, it has been shown that individual 
data packet size can affect the overall communications data rate 
[VA87]. In general, as the data packet size is decreased, the overall 
communications data rate also decreases. This is because the over- 
head time associated with the set-up of a data transmission is more 
significant when the size of the data packet to be communicated is 
small. The current test configuration and software was used to extend 
the scope of testing to further investigate this phenomenon. The 
results of this testing for the T414 and the T800 are shown in Figures 
3.4 and 3.5. These graphs, as well as others in this chapter, were 
conducted with various numbers of links active. The number of links 
active for a particular graph are identified on that graph. Figures 3.4 
and 3.5 show that communications throughput decreases rapidly when 
the packet size decreases below a threshold point of from about 200 
down to 50 bytes per packet. 

Further, note the communications characteristics shown in Figure 
3.5. This graph shows the protocol limited nature of bidirectional link 
operations for the T800 Transputer. Compare the plots for the two 
bidirectional links case with those of the four unidirectional links case. 
In either of these cases, a total of four links are operating. As packet 
size increases from one byte per packet, the plots for both links are at 
first identical. However, as the packet size passes about 20 bytes per 
packet, the two plots begin to diverge. The four unidirectionalgiags 
case data rate continues to increase but the bidirectional link case 


becomes protocol limited and data rate levels off. 
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Figure 3.5. T800 Packet Size Effects on Communication Rate 
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4. Communication Link Effects on Processor Performance 

One of the key questions raised about Transputer perfor- 
mance has been the extent to which communications link activity 
interferes with processor execution speed. There are two ways in 
which such interference can occur. First, both the processor and the 
communication links contend for access to the same internal data bus. 
Secondly, the set-up of a communication and the rescheduling of a 
process upon completion of its communication require some proces- 
sor overhead. Recalling the timing test methodology, where opera- 
tions in a loop were conducted in parallel with a variety communica- 
tions load conditions, timing data is available to address this issue. 

For the purposes of comparing processor performance, the 
average number of loops that were performed per unit time under 
each set of test conditions was taken as the relative measure of target 
processor performance. For each set of test conditions, processor 
performance has been normalized by dividing performance measure- 
ments by the “no communication” performance for that set of test 
conditions. 

Figures 3.6, 3.7, 3.8 and 3.9 present representative results 
from this analysis. As can be seen from these figures, processor per- 
formance drops dramatically as the communications packet size is 
decreased below a certain threshold value. In some cases, perfor- 
Imance was reduced to the point where the processor made no 
progress on its looping calculation; the processor was completely 


communications bound. 
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Figure 3.6. 1414 Performance Degradation with No Loop Operation 





Normalized Loop Performance 
(4 divide operations in loop) 


1000 10000 100000 
Packet Size (bytes) 





Figure 3.7. T414 Performance Degradation 
with Four Divide Operations 
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Figure 3.8. T800 Performance Degradation with No Loop Operation 
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Figure 3.9. T800 Performance Degradation 
with Four Divide Operations 
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In the left-hand region of the performance graphs, where the 
individual data packet size is small, the processor overhead associated 
with the communication is the primary contributor to performance 
degradation. As the data packet size is increased, the time associated 
with actually transmitting the data packet increases. As the actual 
packet transmission time increases, the overhead associated with 
communicating a packet becomes less significant. Therefore, when 
the packet size is very large, any processor performance degradation 
will be due to processor and communications link internal bus con- 
tention. Towards the right-hand side of the performance graphs, pro- 
cessor performance degradation can be seen asymptotically 
approaching this bus contention-only degradation characteristic. 

As can be seen by comparing the different performance 
graphs, the type of calculation performed within the loop also affects 
the amount of performance degradation. Although there is only a 
minimal difference in the overhead limited (left-hand) area of the 
graph, there is a noticeable difference in the bus contention limited 
(right-hand) portion of the graph. In general, the difference in the 
bus contention limited area of the graph is directly related to the fre- 
quency of memory access required by the calculation loop. For short 
instructions (such as assignment) that require frequent bus access, the 
performance degradation is greater than for long instructions (Such as 
division) that require infrequent bus access. Figures 3.10 and 3.11 


illustrate this characteristic. 
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Figure 3.10. T414 Bus Access Frequency Effects 
on Performance Degradation 
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Figure 3.11. T800 Bus Access Frequency Effects 
on Performance Degradation 
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5. Processor Effects on Communications Link Performance 





As a corollary to the question of communications effects on 
processor performance, it is reasonable to ask what effect an actively 
executing process has on the performance of a communications link. 
It might be expected that if the processor is slowed by the effects of 
communication link bus contention, a communication link would also 
be slowed by the effects of processor bus contention. Data was 
extracted from the timing database to determine if such effects did in 


fact exist. A typical sample of this data is shown in Table 3.4. 


TABLE 3.4 
PROCESSOR EFFECTS ON COMMUNICATIONS PERFORMANCE 


Dat ee No Calculation With Calculation 
Packet Size Data Rate Data Rate 
(bytes/packet) | (bytes/second/link) | (bytes/second/link) | 








Table 3.4 shows that the effects of processor activity on 
communication link performance is not significant. The reason for 


this is that, when the communicating process is of an equal or higher 
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priority to the executing process, the communication link has priority 
access to the data bus [In86e]. The communication link is, therefore, 
virtually unaffected by bus contention. Bus arbitration priority was 
defined in this manner so that the lower bandwidth communications 
links would not be idled [Ha8&7]. 
6. Summary 

Through the timing tests that have been performed, it has 
been seen that a strong relationship exists between the calculational 
performance of the Transputer and the operation of its communica- 
tions links. Additionally, the size of individual communications pack- 
ets has a pronounced effect on both the calculational and communica- 
tions performance of the) Tramspuren Knowledge of these 
characteristics can be used by the software designer to develop more 


efficient software. 
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IV. SHARED MEMORY MODEL PROGRAMMING INTERFACE 


A. BACKGROUND 

The concept of Communicating Sequential Processes as imple- 
mented by the Transputer is one methodology for interprocess com- 
munication and synchronization. An alternative concept utilizes 
globally shared memory as a means for interprocess communication 
and synchronization. A body of software and development experience 
that makes use of this shared memory concept exists and could be 
useful in developing programs for a network of Transputers. In 
particular, distributed programs developed in the ADA language 
environment use a shared memory model. 

In a large network of Transputers, the physical connectivity of the 
network is limited by the availability of only four bidirectional commu- 
nications links per Transputer. This limited connectivity often 
imposes restrictions upon the distributed systems software designer. 
The software designer must be aware of and consider the physical 
configuration of the network. As a result, the designer must often 
work around the limitations that the configuration imposes. The use 
of a shared memory model in a network of Transputers is one 
methodology that might be used to isolate the software designer from 
these physical configuration limitations. 

These factors have motivated the development of prototype soft- 


ware to implement a shared memory model environment in a network 
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of Transputers. The purpose of this prototyping effort was to gain 
experience in developing such a system for a distributed system. This 
chapter describes the design, implementation, and evaluation of this 


prototype. 


B EVENT COUNTS AND SEQUENCERS 

When multiple processes are sharing a common resource, as in 
the case of a shared memory system, some means must be available for 
the processes to coordinate or synchronize their use of the resource. 
One means for this synchronization is through the use of Event Counts 
and Sequencers [ReKa79]. 

As its name implies, an event count is a value representing the 
cumulative total number of events of a particular type that have 
occurred in a system. An event count might be used, for example, to 
record and monitor the number of times a particular shared memory 
location has been written to or modified. Many separate event counts 
may be defined in a system for use in monitoring different types of 
events. 

A sequencer is also a value representing a count. This count is 
used to control the sequence in which events occur. Each sequencer 
count value can be thought of as a reservation for a process to use a 
shared resource. These reservations are issued in sequencer count 
order to requesting processes. Processes with reservations are then 
permitted access to the controlled resource in sequencer count order. 
Pen PSTN ENS. Several processes writing to the same memory location 


may use a sequencer to ensure that only one process writes to the 
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location at a time. As with event counts, many Sequencers may be 
defined in a system to control access to different shared resources. 

Several primitive operations are defined for event counts and 
sequencers. These operations are: 


e read(event_count)— The read operation returns the current value 
of a specified event count. 


¢ advance(event_count)— The advance operation increments the 
value of a specified event count. 


¢ await(event_count, count_value)—The await operation suspends 
the process executing the await command until the value of a 
specified event count has at least reached the identified count 
value. Note that execution of the suspended process is not neces- 
sarily resumed immediately when the specified count value is 
reached. The process might not resume until some time after the 
count is reached. 


e ticket(sequencer)— The ticket operation returns the value of the 
next available reservation “slot” for a specified sequencer. 
CG IMPLEMENTATION 
The model of event counts and sequencers was selected as the 
paradigm for implementing the prototype shared-memory model. 
Since a network of Transputers does not physically share any memory, 
the implementation must make use of software to simulate a sharing of 
memory. The core of the implementation is a distributed software 
kernel which executes on each node in the network. Figure 4.1 
depicts the general physical configuration of a network with this 


kernel. 
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The kernel contains the system’s shared memory and maintains 
the system’s event count and sequencer values. The system’s shared 
memory aS well as the event count and sequencer values are appor- 
tioned amongst the kernels operating on different nodes. The kernel 
also “hides” the specific physical configuration of the network from 
the application program by managing all communications between 


program modules and network nodes. Figure 4.2 shows the 
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distributed application program’s view of the kernel. A detailed 


description of the kernel is provided in this section. 
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Figure 4.2. Logical Configuration of Shared Memory Kernel 


The prototype implementation also includes a library of proce- 
dures for interfacing with the kernel. This library includes the basic 
primitive operations defined for event counts and sequencers. In 


addition, primitive operations have been defined and included for 
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accessing shared memory segments. The procedures in this library 
are also listed in this section. 
1. The Kernel 
Figure 4.3 shows a logical diagram of the software kernel. As 
shown, the kernel consists of three major parts: the input/output 
buffers, the communications manager, and the shared memory 


manager. 
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Figure 4.3. Shared Memory Kernel Logical Block Diagram 
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a Input/Output Buffers 
A set of input and and a set of output buffers are provided 
for the hardware communications links. These buffers decouple the 
relatively slow link data transfers from the operation of the kernel. 
The buffers enable the kernel to operate in parallel with network data 
transfers. 
b. Communications Manager 
All communications between a kernel and the library 
interface procedures and between kernels on different nodes are for- 
matted as packets. Two different packet formats are used. One 
format is used for communication between a kernel and a library 
interface procedure and one is used for communication between 
kernels on different nodes in the network. The communications 
manager perform any necessary conversions between the two packet 
formats. Figure 4.4 depicts an example of each type of communi- 
cations packet. A listing of all the packet types is included in 
Appendix C. 


Packet Format Used Between an Inierface Module and the Kernel 









Packet Format Used Between Kernels on Different Nodes 
action | from | from | to to event c c ¢ 
code node id process node id process count id| size ; 

Figure 4.4. Kernel Communication Packet Formats 
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The communications manager processes all received 
packets. If a packet destination is a remote node, the packet is routed 
to the node via the hardware communications link identified by the 
node.link data structure. If the packet is for a local application pro- 
gram module interface procedure, the packet is routed to that inter- 
face procedure via the appropriate local link. Otherwise, the packet is 
processed by the node's shared memory manager. 

c. Shared Memory Manager 

The shared memory manager actually performs the 
operations defined by the various library interface procedures. As a 
result of external requests, this module accesses the kernel data 
structures and, if a response is required, generates appropriate mes- 
sage packets for return to the requesting process. 

2 Data Structures 
Operation of the kernel is best described by examining the 
set of data structures that support the kernel. These data structures 
are central to operation of the kernel. Diagrams of these data struc- 
tures and their interrelation are shown in Figures 4.5, 4.6, and 4.7. 
a node.link 

This array is used to represent the physical configuration 
of a network. Each node in the network is assigned a unique identi- 
fying number. By convention, the assigned numbers start at zero and 
are consecutive. The node.link data structure is a one-dimensional 
array with one element for every node in the network. The value of 


the ith element in the array identifies the number of the Hardy 
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link that is used to send messages to the network node with identify- 
ing number i. This does not necessarily mean that the ith node is at 
the other end of the specified link. Rather, it means that the node on 
the other end of the specified link is the next node in the path to the 
ith node. 

Note that the node.link array may be viewed as one row 
extracted from a type of adjacency matrix that defines the network. 
Such an adjacency matrix array is, in fact, used at compile tirne to 
define the individual node.link arrays. As a result, each kernel hold a 
portion of the overall matrix. Currently, the programmer defines the 
contents of this matrix as the last step of the software development 
process. 

b. count.node 

As was done to uniquely identify nodes, each event count 
and sequencer in the system is assigned a unique identifying number. 
Again, by convention, these numbers start at zero and are consecutive. 
The count.node structure is a one-dimensional array with one element 
for each event count and sequencer in the system. The value of the ith 
element in the array identifies the number of the node that maintains 
the count assigned identifying number i. This array identifies which 
processor in the network is responsible for maintaining each event 
count. 

c. count.array 
This is a two-dimensional array that contains four ele- 


ments for each event count and sequencer maintained by a kernel at a 
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node. The first two of these elements are the current values of the 
event count and sequencer. The third element is either a mil)poimeen 
Or a pointer to the base of a shared memory segment associated with 
the event count and sequencer. The fourth element is eithereama 
pointer or a pointer to a list of processes which have been suspended 
by executing the await library procedure. This list is ordered by value 
of the count value specified when a processes executed the await pro- 
cedure. Each event count and sequencer has a Separate waiting pro- 
cess list associated with it. 
d. count.array.indices 

This is a one-dimensional array that has one element for 
each event count in the system. For counts maintained at a particular 
node, this array contains the index of that count’s data in the 
count.array data structure. For counts not maintained at the node, the 
array contains the nil token. Use of this array speeds access to a 
count's data in the count.array data structure by providing a direct 
pointer to the desired row and avoiding having to search through the 
count.array to find the correct row. 

e. node.awaits 

As was mentioned earlier, associated with each event 
count and sequencer is a list of suspended processes waiting for par- 
ticular count values to be reached. These lists of suspended processes 
are stored using the node.awaits data structure. The data structure is 
a two-dimensional array where each row of the array represents an 


entry for one suspended process. There are four data elements for 
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each entry. The first element of the entry is the count value for which 
the suspended process is waiting. The next two elements identify 
which process at which node has been suspended. The final element 
is either a nil pointer or a pointer to the next entry in that particular 
waiting process list. These pointers and the waiting process pointers 
in the count.array data structure are row index values of the 
node.awaits data structure. Figure 4.5 shows an example waiting pro- 


cess list. 
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Figure 4.5. Kernel Waiting Process List Structure 
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f£ free.list 

This is a one-dimensional array that has one element for 
each array position in the node.awaits data structure. The free.list data 
structure holds a list of node.await data structure row indices that are 
not in use and may be allocated to any event count and sequencer's 
waiting process list. As entries are removed from wait process lists, 
the associated node.awaits row indices are returned to this free list. 

A pointer into the free.list data structure identifies the 
value of the next available node.awaits row index. When this next 
index is allocated, the pointer is incremented to find yet the next 
available node.awaits row index. When an index is returned to the 
free.list data structure, the pointer is decremented and the returned 
index is stored. Figure 4.6 illustrates the use of the free list and asso- 
ciated pointer. 

g. count.size 

This is a one-dimensional array with one element for 
each event count and sequencer in the system. The value of the ith 
element in the array identifies the number of bytes of shared memory 
that are to be associated with count i. 

h. node.data 

This array is the block of shared memory available at a 
node. Segments of this memory block are apportioned based on the 
values in the count.size data structure for counts being maintained at 


that node. The memory pointers in the count.array data structure 


26 


node .awaits 


Wait-for Waiting Waiting Next Await 
Count Node !d Process Id Pointer 


free.list 


t 


Next Free 
Pointer 





Figure 4.6. Kernel Waiting Free List Management 


point to the start of individual shared memory segments in this block 
of memory. Figure 4.7 shows how this shared memory array is config- 
ured and accessed. 
3. Library Procedures 

A functional description of each of the library interface pro- 
cedures is provided below. Each procedure includes a reference to a 
“link” parameter. This parameter is a bidirectional communication 
channel that links or connects a distributed application program 


module to the kernel. The program module does not directly use the 


af 


channel for any communication. The library interface procedures use 
this channel internally for communicating with the kernel. The 


detailed source code for the procedures is provided in Appendix C. 
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Figure 4.7. Kernel Shared Memory Access Data Structures 
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a read(link, count.id, count.value) 

This procedure performs the read operation on the 
specified event count. The value of count.value is set to the current 
value of the count. 

b. advance(link, count.id) 

This procedure increments the value of the specified 
event count. If other processes are waiting on the new value of the 
event count, these waiting processes are resumed. 

c. await(link, count.id, count.value) 

The executing process is suspended until the value of the 
specified count reaches the argument count.value. If the value of the 
specified count is already greater or equal to the argument count.value, 
the process executing the await call continues execution. 

d. ticket(link, count.id, ticket.value) 

This procedure sets the value of ticket.value to the value 
representing the next available reservation for the specified 
sequencer. 

e. put(link, count.id, index, byte.array) 

This procedure provides write access to the shared 
memory segment associated with the specified event count. The array 
of bytes passed to the procedure is stored in the shared memory seg- 
ment offset from beginning of the shared memory segment by the 


number of bytes specified by the value of the index parameter. 
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f£ get(link, count.id, index, byte.array) 

The get procedure complements the put proceduresiam 
providing read access to the shared memory segment associated with 
the specified count. The array of bytes passed to the procedure is 
read from the shared memory segment offset from beginning of the 
shared memory segment by the number of bytes specified by the value 
of the index parameter. The number of bytes read is based on the size 
of the byte.array parameter. 


g. ecs.kernel(link.array,count.node,count.size,node.id, 
node.link) 


The procedure for the kernel is itself included imgiaae 
library. This procedure is executed on each node in the system. The 
link.array parameter for the kernel is an array of all the individual 
links that connect the application program modules at a node to the 
kernel. The other kernel parameters are described in detail in@iie 


data structures section of this chapter. 


D. PROGRAMMING 

This section presents a brief overview of how to program using 
the shared memory interface. A detailed programming example is 
presented in Appendix D. Programming using the shared memory 
interface is accomplished in three basic steps. 

The first programming step is to divide the program into modules 
that should operate in parallel and to define the shared memory seg- 
ments and event counts and sequencers that will be required for 


communication between and synchronization of the modules. This 
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step should performed without considering the physical configuration 
of a network. 

The next step is to code the individual modules using the library 
interface procedures to manipulate the event counts and sequencers 
and to access the shared memory segments. Since this step is also 
accomplished without considering the physical configuration of the 
network, the modules may be coded independently. 

The final step of the software development process is to apportion 
modules and shared memory segments amongst different processors 
in a network. Although any apportionment of the modules and mem- 
ory segments is logically equivalent, this placement process may be 
influenced by practical considerations. For example, one would not 
want to place all the computationally intensive modules on the same 
processor. Further, it would seem reasonable to locate modules shar- 
ing the same memory segment physically close to the network loca- 
tion of the memory segment. To help quantify the factors that may 
influence the network placement of modules and shared memory 
segments, the following section evaluates the performance of the 


shared memory interface under a variety of conditions. 


E. EVALUATION 

To evaluate the prototype programming interface, a set of tests 
was conducted using the interface. The objective of this testing was to 
provide a representative measure of the communications performance 
of a network when using the programming interface. It was desired to 


determine how the network communications performance was 
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affected by variations in certain parameters. This testing utilized 
T414 Transputers operating with a clock speed of 15 MHz and a link 
speed of 20 Mbits per second. These tests are described in the 
following paragraphs. It should be noted that the prototype program- 
ming interface was not optimized for maximum performance. Because 
of this, the interface performance documented here can likely be 
improved. Chapter VI describes some ways in which the program- 
ming interface performance can be improved. 
1. Basic Interface Procedure Timing 

As the first step in examining the performance of the pro- 
gramming interface, it was chosen to measure the execution times of 
the various interface procedures. A test program was developed to 
execute each of these procedures in a loop. The time required to 
execute the loop with each of the interface procedures was measured. 
The time required to execute the loop without an interface procedure 
(i.e., a “null loop”) was measured and subtracted from the interface 
procedure loop times. The resulting time was used to determine the 
average time required for a single interface procedure execution. 

Interface procedure execution times were measured under 
several sets of conditions. Specifically, the network distance between 
an interface procedure and its target event count and sequencer was 
varied and, for the put and get operations, the size of the argument 
data element was varied. The results of this testing are shown in 


Figures 4.8, 4.9, and 4.10. 
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Figure 4.8. Primitive Operation Timing Test Results 
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Figure 4.9. Put Operation Timing Test Results 
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Figure 4.10. Get Operation Timing Test Results 


Figure 4.8 shows two basic types of execution behavior. First, 
the execution times for one group of interface procedures is relatively 
constant over the range of conditions tested. This group of proce- 
dures includes those that do not require any response from the dis- 
tributed kernel. The procedure can pass its communications packet 
to the kernel and the application program module can proceed with- 
out waiting for a reply. Since communication between the interface 
procedure and the kernel is via memory-to-memory transfer, the 
transmission of a communications packet is fast and relatively insensi- 
tive to variations in data element size over the range of kernel opera- 
tion. Note also in Figure 4.9 that the execution time for the put 
operation is significantly less for the zero-hop (Same node) case than 


for the one-hop case. This is because the zero-hop case is executed by 
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performing a memory-to-memory transfer of data as opposed to the 
slower physical link transfer of data in the one-hop case. 

The second type of behavior is for interface procedures that 
do require a response from the distributed kernel. As can be seen in 
Figure 4.8 for the read, ticket, and await procedures and in Figure 
4.10 for the get procedure, execution times are strongly dependent on 
the test conditions. As the network distance is varied, the overall 
execution time increases because the “round trip” communication 
time in the network increases. Further, since data transfer between 
nodes via the hardware links is significantly slower than the memory 
to memory transfer, the effects of data size on execution time for the 
get procedure become significant. 

The results of this testing imply that there is a preferred 
location for the counts and shared memory segment for a producer 
and consumer of data located at different nodes in the network. Since 
the put operation is relatively insensitive to network distance and 
since the get operation is greatly affected by network distance, it 
would appear that the shared memory segment for a producer 
consumer pair should be located at the consumer's node. A test was 
performed to confirm this. In this test, the count and shared memory 
segment for a producer and consumer pair were placed at the 
producer’s node, then at the consumer’s node. The resulting data 
communication rates were measured for various data element sizes. 
The results of this testing are shown in Figure 4.11 for a network 


distance of seven nodes. The results of this test show that the highest 


OS, 


communications rate is obtained when the shared memory segment 
and associated event count for a producer/consumer pair is located at 


the consumer’s node. 
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Figure 4.11. Count and Shared Memory Location Effects 
on Communication Rate 





2 Node Distance Effects on Communication Rate 
The distance between an application program module pro- 
ducing data and the application program module consuming that data 
can be measured as the number of physical links or “hops” in the 
communications path between the node locations of the modules. A 
test was performed to determine the effect that hop distance could 
have on the overall data communication rate between application pro- 


gram modules. In this test, a producer and consumer of data were 
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separated by increasing hop distances. The library interface proce- 
dures were used to transfer data between the producer and consumer 
at the maximum possible rate. In addition, since the size of a commu- 
nication message has a demonstrated effect on performance (Chapter 
III), the size of the data element being communicated using the pro- 
gramming interface was varied. The resulting communications data 


rates are shown in Figure 4.12. 
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Figure 4.12. Hop Distance Effects on Communication Data Rate 





This graph shows that as the hop distance increases, the 
maximum data rate decreases. This decrease is rapid for first few 
hops but less significant as the hop distance is further increased. This 
is because each additional hop represents a proportionally smaller 


increment of in-line delay to the data transfer. As was shown in 
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Chapter III for the Transputer in general, communication of larger 
data elements is more efficient when using the programming 
interface. 

Processes executing on the nodes between the producer and 
consumer may also be affected by the data transfer through a node. To 
examine this, a looping calculation was initiated on a node immedi- 
ately between a producer and consumer pair. The number of loops 
executed per unit time during a data transfer was measured and com- 
pared to the no-data transfer looping rate. Again, varying sizes of data 
elements were used. The results of this testing are shown in Figure 
4.13. 

Figure 4.13 shows that lowest performance of about 70 per- 
cent of normal occurs when using the smallest-sized data elements. 
As the size of the data element increases, the performance increases 
to a maximum of about 80 percent of normal. Performance of the 
intermediate process improves when the data element size is 
increased because the overhead associated with communicating a sin- 
gle message becomes relatively smaller. This is the same general 
effect as was shown in Chapter III for communication with varying 
sized data packets. 

3. Hardware Link Sharing Effects on Communication Rate 

When using the programming interface, it is likely that data 
communication between several pairs of producers and consumers will 
be conducted using the same hardware communications link. In this 


case, the kernel in the node on either end of the link is also involved 
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in the communication. A test was performed to determine what 
effects this link sharing had on the overall data communications rate 
between the nodes. In this test, producers and consumers were 
placed on adjacent nodes. Producers were placed on each node in the 
same number to provide balanced bidirectional utilization of the hard- 
ware link. The number of producers on each node and the size of a 
communication data element was varied. The results of this test are 
shown in Figure 4.14. This plot shows that, although some degrada- 
tion in communications performance does occur due to kernel over- 
head, a substantial data rate can be maintained when a hardware link 


is shared between several users. 
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Figure 4.13. Intermediate Process Degradation 
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Figure 4.14. Shared Hardware Link Communications Rates 


4. Multiple Link Effects on Communication 





In most cases, several links on each Transputer in a network 
will be connected and utilized at any one time. To test the 
communications performance of the kernel under these conditions, a 
varying number of hardware links on one Transputer were 
bidirectionally loaded. The resulting data rates were measured for 
different data element sizes. The results of the test are shown in 


Figure 4.19; 
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Figure 4.15. Multiple Hardware Link Communications Rates 





Figure 4.15 shows that, as the number of links used 
increases, the data actually communicated via each link decreases. 
However, in Chapter III it was shown that link communication rate 
was essentially independent of the number of links active. The kernel] 
must, therefore, the cause of this reduced link utilization. Since all 
data messages must be processed by the kernel and since the kernel 
can only process and transfer a fixed maximum number of messages in 
any time period, a limit on the data rate through the kernel must 
exist. Since Figure 4.15 shows the communication rate per link 
decreasing by more than a factor of two when a second link is acti- 
vated, it appears that the kernel data rate limit is less than the capac- 


ity of a single communication link. Activating additional links, 
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therefore, except to reduce the hop distance between nodesmiau 
network, does not increase the overall communication bandwidth of 
the network. Chapter VI discusses potential changes to the kernel 


which may improve the programming interface performance. 
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V. MESSAGE-PASSING MODEL PROGRAMMING INTERFACE 


A. BACKGROUND 

The shared memory model is one method for isolating the dis- 
tributed systems software designer and programmer from having to 
consider the physical configuration of a network of Transputers. It is 
also possible, however, to utilize a message passing model to provide 
for this isolation. To investigate this alternative, a prototype pro- 
gramming interface based on message passing has been developed. 


This chapter describes and evaluates this prototype. 


B IMPLEMENTATION 

Since the Transputer is based on the concept of Communicating 
Sequential Processes (CSP), using this concept as a basis for a mes- 
sage-passing prototype was a natural choice. Communications in the 
message-passing prototype are, therefore, based on CSP communica- 
tions “rules.” Specifically, message passing is logically point-to-point, 
synchronousm and unbuffered. 

A significant goal in the development of this interface was to per- 
mit application program modules to be written in the Transputer’s 
“native” high-level language, OCCAM, without requiring any additional 
external procedure references. In this way, existing modules pro- 
grammed in OCCAM could be used with the interface. 

As with the shared-memory model prototype, the core of the 


message-passing prototype is a distributed kernel that executes on 


Te 


each node in the network. Figure 5.1 shows the physical configuration 
of such a network. Note that this physical configuration is very similar 
to the physical configuration used in the shared-memory model 


prototype. 
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Figure 5.1. Physical Configuration of Message-Passing Kernel 


Figure 5.2 depicts the logical configuration of the message-passing 


network as seen by the application. The logical configuration of the 
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message-passing network shows individual application program mod- 


ules interconnected by a network of global message-passing channels. 
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Figure 5.2. Logical Configuration of Message-Passing Kernel 


This chapter frequently uses the terms “local channel” and 
“global channel” when referring to the operation of the message- 
passing prototype. A local channel is an actual communications path 
that exists between a process and a kernel. Figure 5.1 shows several 
examples of local channels. Global channels, as shown in Figure 5.2, 
are the virtual communication paths that exist between two processes. 


These virtual communications paths are established and maintained by 
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the distributed kernel. Local channels may be thought of as connect- 
ing processes to the ends of global channels. 
1. The Kernel 

Figure 5.3 shows a block diagram of the message-passing 
kernel. As shown, the kernel consists of three major parts: the 
Input/Output Buffers, Channel Controllers, and the Communications 
Manager. In addition, the kernel performs some initialization of the 
network. Each node broadcasts to the network a list of the global 
channel ends that it has been assigned. This information is used by 
the kernel to create a global routing table for netwoes 
communications. 

a Input/Output Buffers 

These buffers are identical to those used in the shared 

memory model prototype. They decouple the relatively slow link data 
transfers from the operation of the kernel. The buffers enable the ker- 
nel to operate in parallel with network data transfers. 

b. Channel Controllers 

The kernel creates one channel controller process for 

each local channel at a node. The channel controller is the physical 
connection between a local channel and the kernel and is the logical 
connection between a local channel and the appropriate global chan- 
nel. Communication over the global channel is controlled using a sim- 
ple protocol which is managed by these channel controllers. When 
the receiving end of a global channel is ready to receive an application 


program module message, the receiving end channel controller 
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transmits a “ready to receive” message to the sending end controller. 
The sending end controller is then released to accept the applica- 
tion’s message and send it to the receiving application program mod- 


ule via the global channel. 
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Figure 5.3. Message Passing Kernel Logical Block Diagram 
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c. Communications Manager 
As in the shared memory interface, messages communi- 
cated over the network are formatted as packets. Figure 5.4 diagrams 
the packet format used. Each packet received from a hardware link or 
from a channel controller is processed by the communications con- 


troller and routed to its packet header identified destination. 


Packet Format Used Between Kernels on Different Nodes 





Figure 5.4. Kernel Communications Packet Format 


At the current state of the kernel’s development, mes- 
sages transmitted from an application program module over a global 
channel are restricted to only one of the simple protocols predefined 
in the OCCAM language [PoMa87]. The available protocol provides for 
transmission of variable-length byte arrays (INT::[]BYTE). This is not a 
particularly limiting restriction since any structure in OCCAM can eas- 
ily be RETYPED into an array of bytes for communications purposes. 

2. Data Structures 
The message-passing prototype kernel maintains a set of data 
structures that defines the characteristics of the physical and logical 
communications network. The overall operation of the kernel is fun- 


damentally dependant on the interrelation of these structures. The 
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kernel data structures are described in the following paragraphs. 
Figure 5.5 is an example of the manner in which these data structures 
are interrelated at two different nodes. 
a node.link 

This data structure is identical to the node.link data 
structure used in the shared-memory model. The value of the ith 
element in the array identifies the number of the hardware link that 
connects to the next node in the path leading to node i. The array 
defines the node’s view of a network. Currently, the programmer 
defines the contents of this data structure at the end of the software 
development process based on the particular network configuration 
that is to be used. 

b. gchan.node 

This structure is used to identify the end locations of 
network global channels. This is a two-dimensional array which has a 
two-element entry for each global channel defined in the distributed 
system. Each global channel in the network is assigned a unique 
identifying number. By convention, the assigned numbers start at zero 
and are consecutive. This array is indexed by a global channel’s unique 
identifying number. The first element of an entry identifies the node 
location of the process that outputs to the global channel. The second 
element identifies the node location of the process that inputs from 
the channel. During kernel initialization, each node broadcasts a list- 


ing of global channel ends at the node. The kernel uses these 
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Figure 5.5. Kernel Data Structure Interrelation 
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broadcast messages to construct this array. As a result, each node in 
the system has an identical copy of this data structure. 
c. gchan.Iichan 
If a global channel end is connected to a local channel at 
a particular node, this structure identifies which local channel the 
connection is to. This is also a two-dimensional array indexed by a 
global channel’s unique identifying number. Each entry in the array 
consists of two elements. In general, the first element of an entry is 
the identifier of the local channel that outputs to the global channel; 
the second element is the identifier of the local channel that inputs 
from the global channel. At a particular node, this array only includes 
local channel identifiers for the global channels that output from or 
input to that node. All other entries in the array are nil at that node. 
d. loc.chan 
This data structure is an array of OCCAM communication 
channels. This array, potentially different at each node in the net- 
work, defines all the local channels that connect to global channels at 
a particular node. 
e. Ichan.gchan 
This is a one-dimensional array with an element for each 
local channel at a node. The value of the ith element in the array is the 
identifier for the global channel to which local channel i is connected. 
f chan.map 
Each node in a network has a different version of this 


data structure. The data structure is an array that holds two elements 


on 


for each local channel defined at a node. For the it® channel defined 
in the loc.chan array, the it® entry in the chan.map array defines the 
end of the global channel “connecting to” that local channel. The first 
element of an entry is the unique identifier for the global channel. 
The second element of an entry identifies whether the local channel is 
connected to the sending or receiving end of the global channel. 
2. Library Procedures 
The Library for the message passing interface consists only of 


the distributed kernel procedure: 
csp.kKernel(node.id, node.link, chan.map, loc.chan). 


This kernel procedure is executed on each node in the sys- 
tem. In general, the parameters for this procedure identify the physi- 


cal and logical channel mappings for that node. 


CG PROGRAMMING 

This section presents a brief overview of how to develop a pro- 
gram using the message-passing interface. A detailed programming 
example is presented in Appendix F. Programming using the mes- 
Sage-passing interface is accomplished in essentially the same three 
basic steps as is programming using the shared memory interface. 
The first step is to divide the program into modules that should oper- 
ate in parallel and to define the point-to-point channels that will be 
required for communication between the modules. This modulariza- 


tion need not, however, consider the physical four-link limitation of an 


82 


individual Transputer or the actual connectivity of a particular 
network. 

The next step is to code the individual modules using OCCAM 
programming guidelines [PoMa87]. Thus far, the program develop- 
ment methodology is the same as would be used for developing any 
program in OCCAM. 

The final step of the software development process is to apportion 
modules amongst different processors in a network. The same practi- 
cal considerations that influence the placement of channels when 
using the shared memory model also need to be considered when 


using the message passing model. 


D. EVALUATION 

The same types of testing were performed for the message-pass- 
ing interface as were performed for the shared-memory interface. 
This section provides the results of the message-passing model inter- 
face kernel testing and compares the test results of the two interfaces. 
The message-passing interface testing also utilized T414 Transputers 
operating with a clock speed of 15 MHz and a link speed of 20 Mbits 
per second. Since the message passing interface has no separate set 
of procedures for use in application program modules, separate inter- 
face procedure timing tests were not needed. 

1. Node Distance Effects on Communications Rate 

A test was performed to determine the effect increasing dis- 

tance between nodes has on the overall data communication rate 


between application program modules. In this test, a producer and 
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consumer of data were separated by increasing hop distances. Data 


messages were sent from the producer to the consumer at the maxi- 


mum possible rate. The resulting communications data rates for dif- 


ferent data message sizes are shown in Figure 5.6. Figure 5.7 provides 


a comparison of the maximum data communications rates for the two 


interfaces. 
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Figure 5.6. Hop Distance Effects on Communications Data Rate 





Figure 5.6 shows that as the hop distance increases, the 


maximum data rate decreases. This decrease is rapid for first few 


hops, but less significant as the hop distance is further increased. 


This is because each additional hop represents a proportionally 
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smaller increment of in-line delay to the data transfer. This same 


effect was shown for the shared memory Kernel in Chapter IV. 
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Figure 5.7. Comparison of Hop Distance Effects 





Figure 5.7 shows that for smaller network distances, the 
message passing interface has a significant data communications rate 
advantage over the shared memory interface. However, as the net- 
work distance between the producer and consumer nodes increases, 
the communication performance difference between the two 
interfaces decreases. In both cases, minimizing the difference 
between the producing and consuming nodes improves performance. 

With the message-passing model’s greater data communica- 


tion rate, it might be expected that performance of processes on 
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nodes in the path between a producer and consumer would be 
degraded to a greater degree with the message-passing model than 
with the shared-memory model. Figure 5.8 shows the intermediate 


process degradation for both interfaces. 
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Figure 5.8. Intermediate Process Degradation 
During Communication 


As can be seen, the degradation for the message-passing 
model is less than for the the shared-memory interface. The reason 
for this behavior appears to lie in the number of messages an 
intermediate node must process when using the two interfaces. In 
the shared-memory model case, each production/consumption cycle 
requires at least the transmission of three messages between nodes 


(an await message, an advance message, and either a put or a get 
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message). The message-passing model only requires that two 
messages be sent (a receiver ready message and a data message). As 
shown in Chapter III, a greater degradation should be expected when 
using more messages to communicate a given sized data element. 
2. Hardware Link Sharing Effects on Communication Rate 

In the message-passing model, several global channels will 
likely use the same physical link for network communication. As with 
shared-memory interface testing, a varying number of producer/con- 
Sumer pairs were executed on two adjacent nodes to examine the 
kernel’s performance. The results of this test are shown in Figure 5.9. 
Despite Some performance degradation, these results show that a link 


can be effectively shared between global channels. 
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Figure 5.9. Shared Hardware Link Communication Rates 
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3. Multiple Link Effects on Communication Rate 

Testing of the message-passing interface with a varying num- 
ber of links active was performed for the message-passing interface. 
The results of this testing are shown in Figure 5.10. This testing 
revealed the same type of kernel data rate limitation found when 
testing the shared-memory model interface. Although the data rate of 
the message-passing kernel is significantly greater than that of the 
shared memory interface, the message passing kernel’s data rate limit 


remains less than the capacity of one hardware communications link. 
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Figure 5.10. Multiple Hardware Link Communication Rates 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

This thesis has evaluated the performance of an isolated Trans- 
puter and the performance of two abstract programming interfaces 
with distributed kernels for a network of Transputers. The results of 
the evaluation show that, although the use of a distributed kernel 
reduces the effective performance of individual Transputers in the 
network, the performance remains relatively high. In general, the 
performance of the message-passing interface was superior to that of 
the shared-memory model interface. This comparison should not, 
however, be taken as absolute. There are improvements discussed in 
this chapter that can be made to both interfaces which could signifi- 
cantly affect this comparison. 

Thus far, all evaluation of the abstract programming interfaces has 
been based on testing. The programming interfaces also need to be 
examined on the basis of whether or not they can be effectively used in 
the development of distributed programs. By its nature, such an 
assessment is subjective and prone to be influenced by one’s prior 
experience and personal programming style preferences. Based on 
the experience of developing test and example programs for use with 
the interfaces, it appears that both of the interfaces do simplify the 
programming of a physical network of Transputers. The primary fac- 
tor in this is that the programmer is isolated from having to consider a 


specific physical configuration of a network at all stages of program 
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development. Only after the program is developed need a particular 
configuration be considered, at which point the physical configuration 
is described by a set of simple data tables. 

Both interfaces accomplish the goal of isolating the software 
designer from physical configuration considerations. There was, how- 
ever, no clear choice as to which interface methodology was the over- 
all “best” from a programming standpoint. Each methodology had 
certain advantages and disadvantages. In general, the initial design, 
development, and coding of a distributed program was most effectively 
accomplished using the message-passing model. However, once past 
the initial implementation of a program, modifications, either to add 
additional functionality or to correct initial design errors, are usually 
required. In most cases, these modifications and additions were eas- 


ier to make when using the shared-memory model interface. 


B RECOMMENDATIONS 
As was mentioned previously, the programs for these prototype 
interface implementations are not fully optimized. As a direction for 
further work in developing the abstract interfaces for a network of 
Transputers, several modifications to the interface implementation 
should be considered. In general, the same types of changes can be 
made to both interfaces to improve performance. Some suggested 
changes are listed as follows: 
e The message packet format can be improved. The current mes- 
sage format is actually scheduled as three separate transmissions: 
transmission of the header, transmission of a size value, and 


transmission of the data array. Efficiency can be improved by 
reducing the number of required transmissions. For example, the 
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header information and the data array could be combined into a 
single array transmission preceded by a size transmission. This 
would eliminate one scheduled transmission each time a message 
is handled. 


e The transmission of messages within the kernel uses OCCAM 
communications channels. This transmission is performed by 
processor-controlled memory-to-memory transfer. When the size 
of a data message is large, this decreases the performance of a 
node. A simple memory manager could be incorporated into the 
kernel to allocate space for messages in transit so that the trans- 
fer of messages within the kernel could be performed by commu- 
nicating pointers or handles to the messages via the OCCAM 
channels. 


e In the case of the shared-memory model, the interface only 
transfers data from a shared-memory segment to a remote node 
when an application program module at the remote node requests 
the data. If it is known in advance that certain remote nodes or 
modules will require frequent access to a node’s shared memory 
segment, performance could be improved by having the dis- 
tributed kernel automatically and periodically transfer selected 
segments of globally shared memory between nodes. 


e Many of the data structures within the kernels are sparse. As the 
size of a network becomes larger, the storage required for these 
structures will also become larger. At some point, it is likely that 


a compressed storage scheme for the kernel data structures will 
need to be adopted. 


In addition to these performance improvements, the kernel 
should be modified to include a distributed algorithm for examining 
the network to determine a network’s physical configuration. Use of 
such an algorithm would save the software designer from having to 
specify the physical configuration of a network as_ the last step of the 
design process. More importantly, however, to support fault toler- 
ance, such an algorithm could be used to identify an altered physical 
configuration so that alternate message routing paths could be 


established. 
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APPENDIX A 


DETAILED TRANSPUTER TIMING TEST CONFIGURATION 
AND SOURCE CODE 


A. SUMMARY 


The purpose of the timing test configuration and associated test 


code is to provide a general framework for testing Transputer 


performance characteristics. The detailed configuration of the test 


set-up is shown in Figure A.1. The test set-up consists of a central 


“target” Transputer and four “satellite” Transputers, each attached to 


the target Transputer by a communications link. In addition, there 


are associated Transputers to perform the functions of control and of 


data routing and recording. 


B SOURCE CODE 
1. Configuration Section 


ae ee rm ee ee ee ee ee ee ee ee ee ee ee ee Se ee a eae SO ee 


ee SO ane ee SS Se Se a ee ee ee © ee ee ee ee ee ee ee ee ee Se es 


TIMING TEST PROGRAM 


i a a a a Gm ee mm me 2 es ec es ce es mm ee eee Se 


{{{ link definitions 


VAL? \link0outocis. «0: 

VAL © lanklout. (Sas 

VAlc © lankzout. 1S 22 = 

VAL lainkSouteeice.3 - 

VAL link0in tS 4 = 

VAL eink] trike Se 

VAL J¢nk2in Is 6: 

VAL we inka gee ss 

bs 

{{{ declarations 

CHAN OF ANY axm.mid.0, amn.mid.1l, arm mid. Z- Armee se 
mid.amn.0, mid arn, mid.amn.2, Mideamn. 3, 
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Link Numbers satellite 
and Frequencies 
Are as Shown 


: 5 | | a 
echo satellite <——— satellite 
5° [roma] °*'3"° [20 Hg] target [20 Miz] °° 


2 


— satellite 
| 2 


Figure A.1. Detailed Test Configuration 





oe 


amm.echo.0, ammn.echo.1, arm.echo.2, amm.echo.3, 
echo.armm.0, echo.amn.1, echo.amn.2, echo.amn.3, 
host .echo.0, host.echo.2, 
echo.host.0, echo.host.2 : 


a 


{{{ SC target 
...target procedure 
eles 

{{{ SC satellite 
...satellite procedure 


}}} 

{{{ SC echo 
...echo procedure 
b}} 


PLACED PAR 
{{{ echo 0 


PROCESSOR 10 T4 
{{{ channel placements 


PLACE echo.host.0 AT link0Oout : 
PLACE host.echo.0 AT lJlinkOin : 
PLACE echo.arm.0 AT Jlaink3out : 
PLACE axmm.echo.0 AT Jlink3in : 
PLACE echo.amn.1 AT  linklout : 
PLACE amm.echo.1 AT  linklin 


}} 
echo (echo .host.0, host.echo.0, 
echo.amm.0, ammn.echo.0, 
echo.amn.1, amm.echo.1, 0) 
J) 
{{{ echo 2 
PROCESSOR 11 T4 
{{{ channel placements 
PLACE echo.host.2 AT link0Oout : 
PLACE host.echo.2 AT linkOin : 
PLACE echo.armm.2 AT Jlinklout : 
PLACE arm.echo.2 Al saoimkiine.: 
PLACE echo.amn.3 AT 1linkZ2out : 
PLACE arm.echo.3 AT 1link2in 
pit 
echo (echo -host.2, host.echo.2, 
echo.amm.2, amm.echo.2, 
echo amis,” amn.echor3s,. Z) 
2s) 
{{{ satellite 0 
PROCESSOR 13 T4 
{{{ channel placements 
PLACE amm.echo.0 AT link2out : 
PLACE echo.arm.0 AT 1link2in 
PLACE ~armnamad.0 ..AT-jankioucoe 
PLACE mid.arm.0 AT 1linklin 
ae 


satellite (amm.echo.0, echo.amm.0, amn.mid.0, mid.amm.0, 0) 
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hh 

{{{ satellite 1 
PROCESSOR 21 T4 

{{{ channel placements 

PLACE arm.echo.1 AT 1lainkOout : 
PLACE echo.arm.1 AT 1link0Qin 
PLACE arm.mid.1 AT link3out : 
PLACE mid.arm.1 AT 1link3in 
ites 

satellite (arm.echo.1, echo.arm.1, arm.mid.1, mid.arm.1, 1) 
}}} 

{{{ satellite 2 
PROCESSOR 23 T4 

{{{ channel placements 

PLACE arm.echo.2 AT Jlinklout : 
PLACE echo.arm.2 AT linklin 
PLACE arm.mid.2 AT linkZ2out : 
PLACE mid.arm.2 AT 1link2in 
Prt 

satellite (arm.echo.2, echo.arm.2, arm.mid.2, mid.arm.2, 2) 
Ey} 

{{{ satellite 3 
PROCESSOR 12 T4 

{{{ channel placements 

PLACE arm.echo.3 AT link3o0ut : 
PLACE echo.arm.3 AT link3in : 
PLACE arm.mid.3 AT linklout : 
PLACE mid.arm.3 AT linklin 
et 

satellite (arm.echo.3, echo.arm.3, arm.mid.3, mid.arm.3, 3) 
Pt 

{{{ target 
PROCESSOR 20 T4 

{{{ channel placements 


PLACE mid.arm.0Q AT linklout : 
PLACE arm.mid.0 AT Jlinklin : 
PLACE mid.arm.1 AT linkZout : 
PLACE arm.mid.1 AT link2in : 
PLACE mid.arm.2 AT link3out : 
PLACE arm.mid.2 AT link3in : 
PLACE mid.arm.3 AT linkOout : 
PLACE arm.mid.3 AT link0Oin 


}}} 
target (mid.amm.0, arm.mid.0, mid.arm.1, arm.mid.1, 
Mid amine amnemlcd.2, mics arn. 3, ann_mid.3) 


}}} 
2. Target Node Procedure 


dem 0, out. link. We ane link? 1, 
elink-2,;)) Outelink.s, in-Jink: 3) 


ce ee cc cr cr oe > ce cm cm crm cr ce mm cm mc mm ee a i a eS SS SS 


{{{ description 
-- This procedure perfomms communications with adjacent nodes and performs 


Be 


-- calculations in a loop to measure the interaction of cammmication and 
-- calculation. In general, a timer is started and the communications and 
-- the calculation is started. When the communications are camplete, the 
-- timer and the looping calculation are stopped. The time for the 

-- communication and the number of loops performed during this time is 

-- extracted and reported. 


em mee ee i ec es ee ee es es es es es ee ee ee ee ee oe oe 


{{{ declarations 

VAL else IS TRUES: 

VAL one.second IS [(1000000/64, 1000000] 
VAL ave..councy IS, Slr: 


{{{ priority codes 


VAL low TS Os: 
VAL high Loude.< 
bh} 

{{{ speed codes 

VAL slow TSo0ce 
VAL fast dpe ped Oe 


}}} 


{{{ operation codes 


VAL none PS leas 
VAL gybolal Say 
VAL assign TS eee 
VAL add .S ets: 
VAL sub IS:.9 42: 
VAL malt TS oo: 
VAL div Ses on: 
VAL movel00Q IES 7/7 
VAL movel000 IS 8: 


{{{ timedata.tsr 
VAL block.size ams) 100] 01661 
VAL packet ..counts IS [1,2,5,10,20, 50,100) 2007200; 
1000, 2000, 5000, 10000, 20000, 50000, 100000) 
He 


CHAN OF BOOL status : 
CHAN OF INT result : 
es 


{{{ test set 


VAL loop .op iS snus: 
VAL op.count [5 4 
VAL loop sora lo) lows: 
VAL logo. 16e- SiS Lasts: 
VAL Comer 1S high: 
VAL commn..loc ito. fasta: 


SN: 


ns 


PRI PAR 
{{{ do communication and reporting 
{{{ process declarations 


INT trigger : 

INT packet .count, packet.size : 
INT loep. Count; 

INT Start.time, stop.time : 


(block.size]BYTE link.data: 
PLACE link.data AT 4096: 


TIMER clock :; 

aro 

SEQ in.links = 0 FOR 5 

SEQ out.links = 0 FOR 5 
SEQ packet .count .index = 0 FOR SIZE packet .counts 
SEQ 

{{{ initialize for test 
packet .count := packet.counts [packet .count . index] 
packet .size := block.size / packet.count 
}} 
{{{ take care of triggering 
imeiink-0 2 trigger 


PAR 
out.link.0 ! trigger 
out.link.1 ! trigger 
out.link.2 ! trigger 
euc.damk.S |! trigger 
status ! FALSE 


a 
{{{ start the timer 
clock ? start.time 
bi 
{{{ do communication 
IF 
(ap binks: + Our. Links) = (0 
{{{ time one second delay for no links active case 
clock ? AFTER start.time + one.second[conm.pri] 
es 
else 
{{{ set up links 
PAR 
{{{ input links 
{{{ start link 0 input 
IF 
in.links > 0 
SEQ i = 0 FOR packet .count 
in.link.0 ? [link.data FROM 0 FOR packet.size] 
else 
SKIP 
ae 
(iierstare dink 1 input 


oi 


IF 
inelinks 2 
SEQ 1 = 0 FOR packet .count 
in.link.1 ? [link.data FROM 0 FOR packet.size] 
else 
SKIP 
a 
{{{ start link 2 input 
iF 
Drea iene 2 
SEQ i = 0 FOR packet.count 
in.link.2 ? (link.data FROM 0 FOR packet .size] 
else 
SKIP 
}}} 
{{{ start link 3 input 
J83, 
inelinks > 3 
SEQ i = 0 FOR packet.count 
in.link.3 ? (link.data FROM 0 FOR packet.size] 
else 
SKIP 
ae 
ae 
{{{ ouput links 
{{{ start link 0 output 
IF 
out.links > 0 
SEQ 1 = 0 FOR packet.count 
out.link.0 ! (link.data FROM 0 FOR packet.size] 
else 
SKIP 
}}} 
{{{ start link 1 output 
IF 
oul  Janks >i 
SEQ i = 0 FOR packet.count 
out.link.1 ! (link.data FROM 0 FOR packet.size] 
else 
SKIP 
}}} 
{{{ staxt link 2 outpue 
183 
Out links <>) Z 
SEQ i = 0 FOR packet.count 
out.link.2 ! [link.data FROM 0 FOR packet.size] 
else 
2).G0e 
oe 
{{{ start link 3 cuteae 
183 
out links 33 
SEQ i = 0 FOR packet.count 
out.link.3 ! (link.data FROM 0 FOR packet.size] 
else 


Ie 


{{{ stop the loop count 
status ! TRUE 
}}} 
{{{ stop the timer 
clock ? stop.time 
ae 
{{{ get loop count results 
result 2 wiloep -count 
1; 
{{{ report results 
oubelaink,0 ) ave.count 
Guemencn0s | SOOO .ep, Op.count, lloop.pri; loop.loc 
Out link, Om mecmn.or1;, Cam. loc; in, links; out. links 
out Jank.0 4 billock.size; packet .count; packet.size 
out.link.0 ! start.time; stop.time; loop.count 
}}} 
bt} 
{{{ do looping calculation 
{{{ process declarations 


BOOL done : 
INT iteration.count : 
1 By aioe. veo 


[1LO00]BYTE move.data : 
PLACE move.data AT 4096 : 
}}} 
SEQ 
{{{ init calc variables 
b := #FFFFEFFE 
Cc := #FEF 
Hy} 
SEQ in.links = 0 FOR 5 
SFOrour.links = 0 FOR 5 
SEQ packet .count .index = 0 FOR SIZE packet.counts 
{{{ do calculation loop 
SEQ 
iteration.count := 0 
status ? done 
WHILE NOT done 
PRI ALT 
Status ? done 
SKIP 
TRUE & SKIP 
SEQ 
iteration.count := iteration.count + 1 
-- Calulation to be done in the loop goes here 
result ! iteration.count 
vi) 
et 
{{{ COMMENT do no looping calculation 
{{{ do no looping calculation 


oe, 


{{{ process declarations 
BOOL done : 
Ps 
SEQ in.links = 0 FOR 5 
SEQ out.links = 0 FOR 5 
SEQ packet .count.index = 0 FOR SIZE packet.counts 
{{{ do wait for synchronization with communication process 
SEQ 
status ? done 
status ? done 
result ! 0 
ei 
}}} 
ei 


ee ee a ee ea ee me ee ee ee Se oe ee 


PROC satellite (CHAN OF ANY to.echo, from.echo, to.mid, from.mid, 


VAL INT my.tag) 


description 

This procedure provides for sourcing and sinking commmications to place 

a communications load on the target node of the test configuration. If the 
procedure is placed in a the data reporting path from the target node 

to the host system, the satellite also passes along the data from the 
target node. Packet sizes transmitted from this node are the same size 

as the packets being received by the target. 


{{{ declarations 


VAL else 1S TRUee: 

INT trigger : 

INT packet .count, packet.size : 

INT Start.time, stop.time : 

INT ave. Coune : 

INT loop.cp, op. count, loco prijelccp oc: 

INT Comm. pri, Canm.loc, mid.an smi out.. 

INT mid.block, mid.packet.count, mid.packet.size : 
INT mid.start, mid.stop, mid.loop : 


{{{ timedata.tsr 
VAL block.size TS 100000> 
VAL packet.counts IS (1,2,5,10,20, 50, 100, 20Gy sue, 


1000, 2000, 5000, 10000, 20000, 50000, 100000} 


(block.size}]BYTE link.data: 
PLACE link .data AT 4096 : 


100 


TIMER clock : 
}}} 


PRI PAR 
SHO in.links = 0 FOR 5 
SEQ out.links = 0 FOR 5 
SEQ packet .count.index = 0 FOR SIZE packet.counts 


SEQ 
{{{ if 'primary' satellite pass on the trigger 
Te 
my.tag = 0 
SEQ 
from.echo ? trigger 
eoemuc. ! trigger 
else 
SKIP 


J); 
{{{ wait for target start trigger 


from.mid ? trigger 
es 
{{{ initialize for test set 
packet .count := packet.counts [packet .count . index] 
packet .size := block.size / packet.count 
clock ? start.time 
ee) 
{{{ set up links 
PAR 
{{{ start link input 
i 
out.links > my.tag 
SEQ i = 0 FOR packet .count 
from.mid ? [link.data FROM 0 FOR packet.size] 
else 
SKIP 
ey 
{{{ start link output 
105 
ii links > my. tag 
SEQ i = 0 FOR packet .count 
to.mid ! [link.data FROM 0 FOR packet.size] 
else 
SKIP 
et 


i 
{{{ complete test set 


clock ? stop.time 


yy} 
{{{ if ‘primary' satellite pass on the mid results 


IF 
my.tag = 0 
SEQ 
from.mid ? ave.count 
PreneiMicuenlOOO.0p, Op.count, loop.pri; loop.loc 
Premed. 2veconm pri comm. loc; midsin; ~mid.out 


LOR 


from.mid ? mid.block; mid.packet.count; mid.packet.size 
fran.mid ? mid.start; mid-step; midaicos 


to.echo ! ave.count 
to.echo ! loop.op; op.count; loop: pris lcee a= 
to.echo ! conm.pri; conm.loc; midi serene 
to.echo ! mid.block; mid.packet.count; mid.packet.size 
to.echo ! mid.start; mid.stop?; miceiese 
else 
SKIP 


ee 
{{{ report own timing results 
to.echo ! start.time; stop.time 
Jee) 

SKIP 


4. Echoing (Data Routing) Node Procedure 


Se) em me ee eo es ee 2 ee ee es ec © ms 8S ee ee ED ee > et ee ee es Se Se oe ee 


PROC echo (CHAN OF ANY to.root, fram.root, 

to.amn.0, fxam.ann.0, to.amn.l)- tee aan 

VAL INT my.tag) 
{{{ description 
-- This procedure echos timing results from the target and satellite nodes to 
— the host system. The echoing procedure placed in the routing path from the 
-- target to the host echos all the host data. 
rt 


a a a ee i ee ee ce es 2 es ee es ee ee eee ee es Se Se oe ee = aaa 


{{{ declarations 
VAL else [Sieve = 


{{{ timedata.tsr 
VAL block .size tS ..1000cc: 
VAL packet.counts IS [1,2,5,10,20,50,10072707 007 
1000, 2000, 5000, 10000, 20000, 50000, 100000] 


INT ave.count : 

INT loop. 6p, Op.scount;, lesp pri; isepricc : 

INT comm. pri, Comm.loc, mid lin, micwoue.: 

INT mid.block, mid.packet.count, mid.packet.size : 
INT mid start; micestop, midiaicoo 

INT trigger : 

INT arn. Sart, an.Stop; 


eh 
SEQ in.links = 0 FOR 5 


SEQ out.links = 0 FOR 5 
SEQ packet .count.index = 0 FOR SIZE packet .counts 


LOZ 


SEQ 
{{{ if "primary' pass on the start trigger 


BES 
my.tag = 0 
SEQ 
ELC, EOOt ? trigger 
te@. chan” |! trigger 
else 
SKIP 


bh} 
{{{ if ‘primary' echo pass on the target results 


iF 
my.tag = 0 
SEO 
from.amn.0 ? ave.count 
feta Osaeleco.ep; Op.ccunt, 1Gop.pri; loop. loc 
fron.amm.0 ? Ceam.pri; comm.loc; mid.in; mid.out 
from.amm.0 ? mid.block; mid.packet.count; mid.packet.size 
Ercuarm.Ue gma stare, micassteo, mid.loop 
EO-LOOt ! ave.count 
eo rOOu ileep-op, Op count, lcop.pri; loop.lce 
EGUEOO INecumepr!-s conm. lee; mid.in; mid -out 
EO.root ! mid.block; mid.packet.count; mid.packet .size 
EO. EOOE I mid. start, amcesteop, Mideloop 
else 
SKIP 


}}} 
{{{ xveport satellite timing results 


from.armm.0 ? arm.start; amn.stop 


EG. LOOL Porm staly; atm, Stop 
from.arm.1 ? amn.start; armm.stop 
owide. oe | ehaulsisshaers iho camels) 


ea 


PROC root (CHAN OF ANY keyboard, screen, 
[4]CHAN OF ANY from.u.filer, to.u.filer, 
CHAN OF ANY fran.fold, to.fold, 
fromn.filer, to.filer) 
{{{ description 
-- This procedure runs on the host system. It triggers the start of a test 
—- run and gathers data from the network upon completion of the test run. 
—- The data gathered is sent to a disk file on the host system for later 
-- evaluation or transfer. A subset of the gathered data is displayed on 
-- on the host's screen. 


{{{ TDS library references 
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#USE "\tdsiolib\userio.tsr": 

#USE “\tdsiolib\intcere tam. 

ey 

{{{ declarations 

VAL else 1S TRUE 

VAL trigger TSoe ee: 

VAL uS.<per tic IS. p64 ee) 

VAL tab [SS BYte. oe: 

INT id.length, result: 

(63] BYTE id.string: 

CHAN OF ANY data.file: 

{{{ timedata.tsr 

VAL block .size Is” 100000. 

VAL packet .counts IS [1,2,5, 10720720, 100, 200, 2007 
1000, 2000, 5000, 10000, 200007500007 17000a7 

bit 

{{{ network report 

INT ave.count :; 

INT loop.cp, Op .count, lees, or1,7) |copwices, 

INT comm.pri, Camm. loc; mic, mircdeoue =. 

INT mid.block, mid.packet.count, mid.packet.size : 

INT mid.start, mid.stop, mid.loop, mid.tire : 

INT arm.Q.start, ammn.0.stop, ammn.0.time : 

INT arm.l.start, amm.l.stop, arm.1.time : 

INT arm.2.start, amm.2.stco, diru.2-times. 

INT armmn.3.start, amn.3.st0p, damm... 

ve . 

{{{ network channels 

VAL link0out Se 6 ee 

VAL Linklout i 6 ue Oe 

VAL link2out HG -2.e 

VAL link3o0ut IS se: 

VAL link0in IS @4e: 

VAL jemi am IS oe: 

VAL onkZan t6°6 4 

VAL link3in iC ty ae 


CHAN OF ANY from.net.0, from.net.2, 


PAR 
ce 


to.net.0, LOenNeese: = 
to.net.0 AT link2ceue-= 
EOemet a2 AT nic Icut = 


from. net..0 AT link2in 
from.net.2 AT Jank3in 


set the filename and run the host system filer 
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SEQ 
men string FROM 0 FOR 6] := “timed1" 
id.length := 6 
scrstream.to.server (data.file, 
Pron. tt ler te.filer, 
id. length, 1d7string, 
result) 
i 
{{{ run the test network 
SEQ 
SEQ in.links = 0 FOR 5 
SEQ out.links = 0 FOR 5 
SEQ packet .count.index = 0 FOR SIZE packet.counts 
SEQ 
{{{ trigger network 
to.net.0 ! trigger 
a) 
{{{ receive network results 


from.net.0 ? ave.count 

Erem.net 092 loop. co mep. count, slcop.pris loop. lloc 
from.net.0 ? comm.pri; comm.loc; mid.in; mid.out 
from.net.0 ? md.block; mid.packet.count; mid.packet.size 
from.net.0 ? mid.start; mid.stop; mid.loop 

from.net.0 ? amm.0.start; ammn.0.stop 

from.net.0 ? amm.1l.start; amn.1.stop 

from.net.2 ? amm.2.start; ammn.2.stop 

from.net.2 ? armm.3.start; amn.3.stop 


mid.time := (mid.stop-mid.start) *uS.per.tic[comm.pri] 


amm.0.time := ammn.0.stop-arm.0.start 
armm.1.time := arm.1.stop-armm.1.start 
am.2.time := armm.2.stop-amm.2.start 
amm.3.time amm.3.stop-ammn.3.start 
i 

{{{ put results to file 

{{{ processor type 

write.int (data.file, 800, 1) 
write.char (data. file, tab) 

el) 

{{{ processor speed 

write.int (data.file, 20,1) 

write.char (data. file, tab) 

ye 

(iq dbink "specd 

write.int (data.file,20,1) 

write.char (data. file, tab) 

ei 

{{{ loop conditions 

write.int (data.file, loop.op, 1) 
write.char (data. file, tab) 

write.int (data.file,op.count, 1) 
write .char (data. file, tab) 

write.int (data.file, loop.pri, 1) 


LOS 


write. 
write. 
Wiis 


}}} 
tte 


ee 
att 


write 


write 
bt 
la 


write. 
write. 
write. 
write. 


1h} 
at 


write. 
.char (data. file, tab) 
write. 
.char (data.file,tab) 
write. 
write. 
write. 


write 


write 


char (data .file, tab) 
int (data.file, loop.loc, 1) 
char (data. file, tab) 


link conditions 
write. 
write. 
write. 
write. 
write. 
write. 
write. 
write. 


int (data. file, ccompra, | 
char (data.file, tab) 

int (data.file,comm.loc,1) 
char (data.file, tab) 

int (data £116 mice) 
char (data.file, tab) 

int (data.file,mid.out,1) 
char (data. file, tab) 


cam. conditions 

int (data file miceslock 
write. 
write. 
write. 
write. 
char (data. file, tab) 


char (data. file, tab) 
int (data.file,mid.packet.count, 1) 
char (data .file, tab) 
int (data. file,mid. packet .size, 1) 


target results 


int (data. file,mid.loop, 1) 
char (data.file, tab) 
int (data. file,mid.tire, 1) 
char (data.file, tab) 


satellite results 


int (data. file,amn.0.tine, 1) 
int (data.file,amn.1.tire, 1) 
int (data.file,armm.2.tine, 1) 


char (data.file, tab) 
int (data.file,amn.3.tine, 1) 


newline (data.file) 


aD 
Phe 
ae 


write. 
write. 
write. 
write. 
write. 
write. 
write. 
write. 
write. 
write. 


put results to screen 


Ine Links 3) 

out . links, 3) 
mid.packet .count, 7) 
mid.packet .size, 7) 
mid.loop, 9) 
mid.time, 9) 
amm.Q0.time, 9) 
arm.l.time, 9) 
armm.2.time, 9) 
amm.3.timre, 9) 


int (screen, 
int (screen, 
int (screen, 
int (screen, 
int (Screen, 
int (screen, 
int (Screen, 
int (screen, 
int (screen, 
int (screen, 


newline (screen) 


eat 


{{{ terminate host system filer 
data.file ! 24 
Oe 
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10m 





APPENDIX B 
TIMING DATABASE DESCRIPTION 


A. DISCUSSION 

The large volume of data generated by individual timing tests was 
loaded into a database management system to facilitate ease of access 
and aggregation of any future test results. The Ingres Relational 
Database Management System, currently available on the departmental 
mini-computer, was Selected for this purpose. The Ingres database 
management system provides a wide range of tools for accessing and 
manipulating data. These tools range from a database query language 
that can be used to program complex data manipulations and opera- 
tions to a completely menu-driven interactive shell for accessing the 
database. 

This appendix documents the format of the Ingres timing 
database. Also, Some examples are provided on ways in which timing 
data can be accessed through the use of the query language. Complete 
documentation for the use of the many features of the Ingres relational 


database system may be found in [Re85]. 


B DATABASE DESCRIPTION 
A relational database consists of a set of relations. A relation is, in 
turn, composed of a set of data fields. A relation can be thought of as a 


table where the data fields represent the columns of the table. One 
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row of the table then represents one individual set of data entered in 
the table. 

In the timing database, there are several relations that are 
defined. The first of these relations, shown in Table B.1, is the rela- 
tion that actually holds the timing test data. This table shows each of 
the fields in the timing relation, the type of data the field is formatted 
for and the length of the data field (in bytes). 


TABLE B.1 
TIMING DATA RELATION DEFINITION 


Relation Name: timing 


column name type length 


processor integer 
procspeed integer 
linkspeed integer 


opcode 
Secount 
ifeOopprEL 
loeploe 
commpri 
commloc 
iioeel.acnmel 
linksout 
blocksize 
packcount 
packsize 
rseoo count 
looptime 
armOtime 
armltime 
arm2Ztime 
arm3time 


integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
integer 
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The name of this relation is “timing.” Note that the fields defined 
in this relation are the same as the data elements written to the host 
system file by the timing test program of Appendix A. 

There are two additional relations in the timing test database. 
These relations, shown in Tables B.2, B.3, and B.4, provide for trans- 
lating the numerically encoded calculation loop operation codes, the 
timing test priorities, and memory locations in the “timing” relation 


into a text description of the corresponding test condition. 


TABLE B.2 
RELATION DEFINITION FOR CONVERTING 
ENCODED OPERATION CODE NAMES 


Relation Name: opcodenames 


column name type length 


opcode integer 1 
opcodename character 2 





TABLE B.3 
RELATION DEFINITION FOR CONVERTING 
ENCODED PRIORITY NAMES 


Relation Name: prinames 


column name type length 


pew integer 1 
priname character SZ 
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TABLE B.4 
RELATION DEFINITION FOR CONVERTING 
ENCODED LOCATION NAMES 


Relation Name: locnames 


column name type length 


LOC integer ab 
locname character 3Z 





C EXAMPLES 

There are three database operations that are provided as exam- 
ples of query language interaction with the database. These operations 
are: loading an external text file of data into the database, retrieving 
information from the database, and, finally, transferring data from the 
database to an external text file. 

1. Data Loading 

Figure B.1 is a listing of the query language instructions for 

transferring a file of test data from the timing test program into the 
timing database. For the most part, these instructions are self 
explanatory. Of note are the formatting codes, “cOtab” or “cOnl,” 
associated with each of the fields. These formatting codes mean that 
the text file representation of of a field’s data is a variable length char- 


acter string terminated by a tab character or by a new line character. 


Copy "Timing 


processor 
procspeed 
linkspeed 
opcode 
opcount 
l6ospEx 
loop lGe 
commpri 
commlGe 
linksin 
linksome 
blocksize 
Dackeoune 
packsize 
loopcount 
looptime 
armOtime 
armltime 
arm2time 
arm3time 
) 


from "/work/ ingresy cimee ene 


e0taby 
ecU tab 
c0tab; 
CcUGdab, 
e0tas; 
G0tab- 
cOtab? 
Suiab, 
COtao-, 
clita, 
COtals, 
cOtals- 
e0tab,; 
cOtap,; 
C0tab, 
cOtab, 
cOtab, 
cOtab, 
cOtab, 
Sone 





2. Data Retrieval 

Figures B.2 and B.3 are examples of query language data 
retrievals. In Figure B.2, the time (in wseconds) for single link bidi- 
rectional communication of 100,000 bytes with a packet size of 10 
bytes is retrieved for all types of processors. The results of this 
retrieval are also displayed. 

In Figure B.3, the same retrieval is processed but, instead of 
displaying the results, the results of the query are used to form the 


new relation “new_relation.” 


retrieve 
( 
timing.processor, 
enmabiglere Ike eyone abel 
) 
where 
eAuuiorey, (babocc abo. 1 
Eamiric. kath sour 1 
timing.packsize 156 
Enric eeeoue opcodenames .opcode 
opcodenames .opcodename ‘SoU dhe dKeyo) om 
sort by 
Epmang. processor, 
timing.looptime 


Executing 
|proces|looptime 


nS = S| 
LAS 14 3 | 





Figure B.2. Example of Retrieval for Display 


Bor evel mine enew Celar ion 
( 
Mina pmoCcesSSOL, 
timing.looptime 
) 
where 
IC ILM ake) SAL Ate etoyalh @) 1 
1 
10 
opcodenames.opcode 
STL evoysyy 


Pine soln kSout 

timing.packsize 

EMC Opeode 

opcodenames.opcodename 
SOrt, Dy 

timing.processor, 

timing.looptime 


TS 





Figure B.3. Example of Retrieval for Forming a New Relation 
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As can be seen in Table B.5, which lists the characteristics for 
this new relation, the fields in the new relation inherit their charac- 


teristics from the relation from which the field originated. 


TABLE Bre 
RELATION DEFINITION FOR THE RETRIEVED NEW_RELATION 


Relation Name:  newsge tore 


column name type length 


processor integer 2 
looptime integer 4 





3. Data UnLoading 
Figure B.4 shows the query language listing for transferring 


the previously created relation “new_relation” to an external text file. 
As with the loading of data into the database, a formatting code is 
associated with each field to be transferred to the external text file. 
The interpretation of these formatting codes is the same as for the 


formatting codes for loading the database. | 


copy new rnelatuen 

( 

processor cOtab, 
looptime e0msl 





into "/work/ingres/ textenies 


Figure B.4. Query Language Listing for Unloading the Database 


me PP EINDLX C 


DETAILED SOURCE CODE FOR THE SHARED MEMORY ABSTRACT 
INTERFACE LIBRARY 





A SYMBOLIC CONSTANTS 


{{{ LIB this is LIBRARY "\ecslib\ecssymb.tsr" 
ae Library ID 

{{{ VAL Declarations 

{{{ useful symbolic constants 


VAL ELSE IGS CY ANSI GO} oh 
VAL NIL Se SI 8 
VAL OUTPUT ie, (0) 8 
VAL INPUT WS eye 


b}} 

{{{ definition of system limits 

-- define the maximm size of an individual message 
VAL MAX .MESSAGE ..SIZE is 10ze: 


-- define the maximm size of one node's part of the 
-- globally shared memory 
VAL MAX .DATA.PER. NODE IS 4096 : 


—~ define the maximm number of event counts allowed 
-- in the system 
VAL MAX .COUNTS IS 256 


-- define the maximm number of event counts to be 
-- maintained by one node 
VAL MAX .COUNTS . PER.NODE i One. 


-- define the maximm space allocation for the waiting 
—~ process lists 
VAL MAX. AWAIT.QUE.ENTRIES IS 16: 


-- define the maximm number of processes per node 
VAL MAX. PROC .PER.NODE LS Ceo... 


{{{ action codes for communications packets 


VAL READ .CMD TSO ae 
VAL READ . REP ES eae 
VAL AWAIT . CMD 1S 20: 
VAL AWAIT. REP TS eels: 
VAL AWAIT .QUE. FULL BES: i 7 
VAL ADVANCE .CMD IES See, 5 8 aes 
VAL ADVANCE . REP IES Os ee 


eS 


VAL 
VAL 
}}} 
}}} 
}}} 


B KERNEL PROCEDURE 


TICE TCD 
eR Ree 


GET. 
CET REP 


PUT. oD 
BUI RES 


16S) 
LS 


IS 
IS 


JES 
IS 


40 : 
41 : 


50> 
oy es 


CU: 
Gdn: 


ames 430 a So ee oe oe ee ee ee ee se ee en ee a ee ee es ss es es ee ee a ee ee ee 


PROC ecs.kernel ([] [2]CHAN OF ANY links, 


VAL INT 


VAL [] INT 
VAL []INT 
VAL [] INT 
{{{ description 
-~ Event Counts and Sequencers for the Transputer 


Moee. ic, 
count .node, 
count .size, 
node. 1ink) 


-- This program is an distributed kernel for implementing event counts and 
-- sequencers on a network of transputers. It should be run at high 

-- priority in parallel with application program modules. The application 
-—- program modules are linked to the kemel via bidirectional communication 
-- channels that are elements of the links channel array parameter. 


}}} 


fl 


libraries 


#USE "\ecslib\ecssymb.tsr" 


oe 
{{{ 
eS | 


local declarations 
local abbreviations 


VAL INT num.nodes is SIZE node.link : 

VAL INT num.counts is SIZE Count nedes: 

VAL INT no.links IS SIZE links : 

VAL INT no.h.links rs a 

VAL INT no.s.links IS no.links - no.h.links : 


}}} 

{{{ channel declarations and assignments 

— channels for kermel control of the hard/physical communications links 
{2*no.h.links]CHAN OF ANY hard stinks: 

PLACE hard.links AT Ors 

}}} 

{{{ communications data structures 

— basic fommat of a received cammmnications header 

[6] INT header : 


[] INT short .neader IS {header FROM 0 FOR 2] 
INT action.code rs header [0] 
INT Count ac ES header [1] 


WS 


INT to .node ES header [2] 


INT COnPLOC aS header [3] 
INT from.node tS header [4] 
INT £ram. proc ins header [5] 
INT data.size :; 


— potential fommats of received data arrays 
[MAX .MESSAGE.SIZE]BYTE data.array : 


INT count.value RETYPES ([data.array FROM 0 FOR 4] 
INT index RETYPES ([data.array FROM 0 FOR 4] 
INT seg.size RETYPES ([data.array FROM 4 FOR 4] 


re} 
{{{ event count data structures 


— variables to track allocation of shared resources 
INT node.counts, 

base.count, 

next .free : 


-- storage allocation for kernel data structures 


[MAX . COUNTS ] INT count .array.indices : 
[MAX.COUNTS .PER.NODE] [4] INT count.arrey : 

[MAX .DATA.PER.NODE] BYTE node.data : 

[MAX .AWAITT .QUE.ENTRIES] [4] INT node.awaits : 

[MAX .AWAIT .QUE.ENTRIES] INT free. 1st. 


}}} 

ae 

{{{ procedures 

{{{ PROC buffer.in.link(chan.in, chan.out) 


_—— > aS > oe ee eee ee SE Se es es ee ee ee ee eC ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee ee 6 er en ane en ae ae ee ewer oe ee er ee oe ee Se Se ee eee 


PROC buffer.in.link (CHAN OF ANY in.link, out.link) 


oD ED ams a a <a me ee ee a ee ee ee ee es ee eS ee ee es ee ee es ee ee aS SS SS SS SS 


{{{ description 

— This procedure provides a 'soft' buffer for hard link input and output. 
— This buffer increases the overall throughput of the node. 

an 





{{{ declarations 


[MAX .MESSAGE . SIZE] BYTE data.array.0, 
eriwsleleiasive di - 
[6] INT header .0, 
header.1 : 
INT data.size.0, 
data.size.1l: 
}}} 
SEQ 
in.link ? header.0; data.size.0::data.array.0 
WHILE TRUE 
SEQ 
PAR 
out.link ! header.0; data.size.0::data.array.0 
in.link ? header.1; data.size.1::data.array.1 
PAR 
out.link ! header.1; data.size.1::data.array.1 


alee 


in.link ? header.0; data.size.0::data.array.0 


{{{ PROC buffer.out.link (chan.in,chan.out) 


ee a ee ae a ee ee es es es ee ee es ee ee ee es ee es ee es es ee es es eee i ee ee 


PROC buffer.out.link (CHAN OF ANY in.link, out.link) 

{{{ description 

— This procedure buffers output to hardware communications links. 
— To improve perfommance, the buffer is sized to hold one message 
— for each process at the node. The buffer is organized in a ring 
— configuration. WARNING: This implementation uses variables 

— shared between parallel processes. The ring buffer itself, and 
— next in and out pointers are shared. When this procedure is mn 
— at high priority, the sequencing of the code guarantees that 

— there will be no access conflicts to these shared structures. 

— The purpose for this sharing is that the implementation is 

— slightly faster under average case loading conditions and is no 
— worse than a 'nommal' request-next-item buffer under any conditions. 


{{{ declarations 

-- a local alternate name for the system 11,it used to 
-- size the buffer 

VAL INT buff.size igs) MAX.PROC.PER.NODE : 


-- pointers to positions in the ring buffer 
INT Hex .27, 
next.out : 


-- define the ring buffer 

[buff.size}][6]INT header : 

[buff .size] INT data.size : 

[buff.size] [MAX.MESSAGE.SIZE]BYTE data.array : 


-~ channels for communicating between the input and output 
=< parts o£ the buffer 


CHAN OF ANY wake.in, 
wake.out : 
}}) 
SEQ 
{{ {initials zat16n 
HExXt an ad 


next.out := 0 
I 
PAR 
{{{ buffer input 
{{{ local declarations 
INT Butteno;, 
any : 


ie 


118 


WHILE TRUE 
SEQ 
buff.no := next.in\buff.size 
PRI ALT 
— if the buffer is not full 
(nmext.in < (next.out + buff.size)) & SKIP 
SEQ 
— input an item to the buffer 
in.link ? header[buff.no]; data.size[buff.no]::data.array [buff.no] 
next.in := next.in + 1 
-- if the buffer was empty, let the sleeping output know 
IF 
(next.in - next.out) = 1 
wake.out ! NIL 
ELSE 
SKIP 
-- the buffer is full, go to sleep until item output 
wake.in ? any 
SKIP 
ee 
{{{ do output 
{{{ local declarations 
INT Pures], 
any : 
}}} 
WHILE TRUE 
SEQ 
buff.no := next.out\buff.size 
PRI ALT 
-- if the buffer is not empty 
(next.in > next.out) & SKIP 
SEO 
—~ output a buffered item 
out.link ! header [(buff.no]; data.size[buff.no] ::data.array [buff .no] 
next.out := next.out + 1 
-- if the buffer was full, wake the sleeping input process 


IF 
(next.in - next.out) = (buff.size - 1) 
wake.in ! NIL 
ELSE 
SKIP 


-- the buffer is empty, wait for a wake-up after some input 


wake.out ? any 
SKIP 


cm mm ccm mt mm cm crm me ce ce cr cm cc cr cr cc cc mc ee ee ee a ee Se Se SS SS 


{{{ PROC send.packet (header, data.size, data.array) 


PROC send.packet (VAL [6]INT header, VAL INT data.size, []BYTE data.array) 


ee ce ee es ee ee ee es Se ee es ee se ee ee ss ec et ee = i ee es SS mS eS 


{{{ description 
—— This procedure packages messages for sending either to a remote node or 


PAS 


— to a local process. 
EE 


{{{ declarations 
-—— define subset of overall header for local communication 


VAL [JINT  short.header IS [header FROM 0 FOR 2] 
VAL INT to .node is header [2] 
VAL INT to.proc is header [3] 
Bu 
IF 
{{{ packet is to local procedure -- retum it locally 


to.node = node.id 
links [to.proc] [OUTPUT] ! short.header; data.size::data.array 
bh} 
{{{ else packet is for remote node -- pass it on 
ELSE 
links [node .link[to.node]] [OUTPUT] ! header; data.size::data.array 


{{{ PROC process.packet (link .no) 


ee se ee ee oe ee ee i ew ae ee ee ee ee ce ee es es ee ac ee ee 2 es es ee ee a ee ee ee ce ee es 


PROC process .packet (VAL INT link.no) 

{{{ description 

—— This procedure perfomms a function based on the value of the action.code 
— element in the header of a cammmnications packet. These functions 

— correspond to the basic calls provided by the event counts and sequencers 
— procedures (read, advance, await, ticket, put and get). 


{{{ command packet requires processing by my node — handle it 
(to.node = node.id) AND (to.proc = 0) 

{{{ get characteristics of count.id 

INT count. index IS count.array.indices [count .id] 

INT current.count IS count.array([count.index] [0] 

INT current.ticket IS count.array[count.index] [1] 


[ 
INT base IS count.array[count. index] [2] 
INT head TS count.array [count . index] [3] 
bh} 
IF 


{{{ read command 
action.code = READ.CMD 
-~ represent the count as an array of bytes 
[JBYTE out.count RETYPES current.count : 
-—- return the value of the count 
send.packet (([READ.REP, count.id, from.node, from.proc, node.id, 0], 
SLZE OUR GOUNnE, OWE count) 


i 
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{{{ advance command 
action.code = ADVANCE.CMD 
{{{ declarations 


-- Each time an advance is perfoured, the waiting process list 

-- associated with the target event count is checked to detemmnine 
-- if the advanced count is a waited-for count. If so, a wake-up 
—- message is sent to the suspended process. 


BOOL more.to.wake: 


}}} 
SEQ 


{{{ initialization 
-—- advance the event count 
Current coum: == Current.-count + 1 


more .to.wake 


de 


:= TRUE 


WHILE more.to.wake 


i 


{{{ mo items on the waiting process list 
head = NIL 
more.to.wake := FALSE 


ie 


{{{ check the first item on the waiting process list 


ELSE 


{{{ abbreviations 
— extract the wating process list entries for the first item 


ae cle 
INT 
INT 
INT 
INT 
b}} 
IF 
Li 


the waiting process list. 

wait.count IS node.awaits [head] [0] 
wait.node IS node.awaits [head] [1] 
wait.proc IS node.awaits [head] [2] 
wait.next IS node.awaits [head] [3] 


count reached - wake the first process on the list 


wait.count <= current.count 
SEQ 


node.id, 0], 


fast 


ie 


a 


{{{ send wakeup message 
send.packet ([AWAIT.REP, count.id, wait .node, wait.proc, 


QO, data.array) 


eh 
{{{ remove the first item, return array position to free 


next.free := next.free - 1 
free.list [next.free] := head 
head := wait.next 

ee 


count not reached - no more to wake 


ELSE 
more .to.wake := FALSE 


ei 


Pau 
ited 


eal 


{{{ 


await command 


action.code = AWAIT.CMD 


nodeiid, 2617 


ise 


185; 


{{{ wakeup if count already reached 
count .value <= current.count 
send.packet ([AWAIT.REP, count.id, from.node, from.proc, node.id, 0], 


}}} 


QO, data.array) 


{{{ add to waiting process list if a future count 


ELSE 
jO2 


{{{ mo room for an await list entry - send back await fail 
next .free = MAX.AWAIT.QUE.ENTRIES 
send.packet ([AWAIT.QUE.FULL, count.id, from.node, from.proc, 


}}} 


QO, data.array) 


{{{ is room for an await list entry - add the entry to the await 


ELSE 
1a 


abbreviations 


— extract and abbreviate the await list entries for the 
— next available position (from free list) 


wait.count IS node.awaits([free.list [next .free] ] [0] 
wait .node IS node.awaits [free.list [next .free] ] [1] 


wait.proc IS node.awaits[free.list [next .free] ] [2] 
wait.next IS node.awaits[free.list [next .free]] [3] 
{{{ handle list empty case 
head = NIL 
SEQ 


wait .next := NIL 
head := free.list [next .free] 
}}} 
{{{ handle insert at head of list case 
count.value <= node.awaits [head] [0] 
SEQ 
wait.next := head 
head := free.list [next .free] 
}}} 
{{{ handle all other cases 
ELSE 
{{{ declarations 
— walk the list to find the proper ordered insertion point 
—- following local variables track position of the search 
INT CUuESOG, 


joygnlloyel op 
3} 
SEQ 
{{{ initialize 
prior := head 


cursor := node.awaits [head] [3] 
bia 


Ne 


{{{ sean list 
WHILE cursor <> NIL 


i103: 
count .value <= node.awaits[cursor] [0] 
cursor := NIL 
ELSE 
SEQ 
MELOL, -— Clirsor 
cursor := node.awaits [cursor] [3] 


}}} 
{{{ insert into list 


wait .next := node.awaits([prior] [3] 
node.awaits [prior] [3] := free.list [next .free] 
later 


ie 
{{{ build table entry - remove position from free list 


wait .count := count .value 
wait.node := from.node 
wait.proc := from.proc 
next.free := next.free + l 


ty) 
ait 
1) 


re) 
{{{ ticket command 


action.code = TICKET.CMD 
—- transform ticket value to byte array for transmission as data 
[]JBYTE out.count RETYPES current.ticket : 
-—- send packet and increment the ticket value 
SEQ 
send.packet ([TICKET.REP, count.id, from.node, from.proc, node.id, 0], 
SIZE TOuUuE.CcOuUnL, GUE.COUunt) 
current .ticket := current.ticket + 1 
pe) 
{{{ put cammand 
action.code = PUT.CMD 
{{{ local definitions 
-- define the location in the node.data array of shared memory 
-- for where to write the transferred data 


VAL INT put .base rs base + index : 

VAL []JBYTE index.array RETYPES index :; 

VAL INT index.size IS SIZE index.array : 

VAL INT put.size ius data.size - index.size : 


He 
—=- store the requested data array 


[node.data FROM put.base FOR put.size] := 
[data.array FROM index.size FOR put.size] 
ey 
{{{ get canmmand 
action.code = GET.CMD 
-- determine the starting location for the read operation 
-- in the shared memory segment 
VAL INT get.base IS base + index : 
-- send the requested data 
send.packet ([GET.REP, count.id, from.node, fram.proc, node.id, 0], 
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seg.size, [{node.data FROM get.base FOR seg.size}) 
b}} 
se) 
{{{ else -- just pass the message on to its destination node 
ELSE 
send.packet (header, data.size, data.array) 
bie 


hae 
{{{ PROC build.packet (link.no) 





PROC build.packet (VAL INT link.no) 

{{{ description 

— To reduce the internal node communication load, message headers from local 
— processes include only a subset of the elements used for communicating 

—- between nodes. This procedure 'expands' a local commmications packet 

— header into an external packet header. 

ber 


SEQ 

from.node := node.id 
fuem.proc := link.no 
to.node = count .node [count .id] 
EO. SLCe = 0 

lat 

Pe 

SEQ 


{{{ > inwtiali zation 
{{{ initialize the count.array and count.array.indices 
SEQ i = 0 FOR MAX.COUNTS 

count .array.indices[i] := NIL 


node.counts := 0 


base.count := 0 
SEQ 1 = 0 FOR num.counts 
TF 
count .node[i] = node.id 
SEQ 
—- enter quick look-up index 
count .array.indices{i] := node.counts 
— build initial count data 
count .array [node.counts] [0] := 0 


count .array {[node.counts] [1] 0 

count .array {node.counts] [2] base.count 
count.array([nede.counts] [3] := NIL 

-- allocate shared memory segment 
node.counts := node.counts + 1 
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base.count := base.count + count.size([i] 
ELSE 
SKIP 
tiie! 
{{{ initialize the free list 
SEQ 1 = 0 FOR MAX.AWAIT.QUE.ENTRIES 
free.list[{i] :=1 
next .free := 0 
ey 
}}} 
{{{ run system 
PAR 
{{{ buffer hardware links 
PAR link.no = 0 FOR no.h.links 
PAR 
buffer .in. link (hard. links [link.no + no.h.links], links [link.no] [INPUT]}) 
buffer.out..link (links [link.no] [OUTPUT], hard.links [link.no]) 
}}} 
{{{ monitor cammunications 
{{{ local declarations 
-- The communications monitoring procedure uses a 'fair' implementation 
-- of the ALT structure. In general, it provides that if a communication 
-- was just received from one of several channels, that channel will have 
-- the lowest priority for the next execution of the ALT. In this way, 
-—- no single cammmications channel cam 'starve' access to the kernel 
-- from the other cammunications channels. The variables defined 
-- below are used to track the last communicating channel for this 
-- ‘'fair' ALT. Note that hard and soft links are treated separately. 


INT last.h, last.s: 
}}} 
SEQ 
{{{ initialization 
last.s := 0 
last.h := 0 
ra 
WHILE TRUE 
PRI ALT 
{{{ handle external hardware links 
ALT link.no = 0 FOR no.h.links 
links [( (link.no + last.h)\no.h.links] [INPUT] ? header; 
data.size::data.array 
SEQ 
process .packet ((link.no + last.h) \no.h. links) 
ljast.h := (no.h.links - 1) + link.no 


Lit 
{{{ handle local soft links 
ALT 1 = 0 FOR no.s.links 
links [((1 + last.s)\no.s.links) + no.h.links] [INPUT] ? short .header; 
data.size::data.array 
VAL INT Jlink.no IS ((i + last.s) \no.s.links) + no.h-.links : 
SEQ 
build.packet (link .no) 
process .packet (link .no) 
last.s := i+ (no.s.links - 1) 


2 


Oe 
ae 
Jae 


CG READ PROCEDURE 
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PROC read({2]CHAN OF ANY link, 
VAL INT Geune.1d, 
INT count . value) 
{{{ description 
-- This procedure reads and returms the value of the argument specified 
-- event count. 


oe es ee ee ee es es cs ee ms ee mm mcm me ae ee ee i O's eS ee oe 


{{{ libraries 
#USE “\ecslib\ecssymb.tsr" 
ae 


{{{ declarations 
— altemalte channel names 


CHAN OF ANY linkin tS ink On 
CHAN OF ANY lLinkout [Sy olankt) 
— caommmications data structures 
[2] INT short .neader : 
INT data.size : 
(] BYTE data.array RETYPES count.value: 
9 
SEQ 
-—- request count value from kernel 
linkout ! [READ.CMD, count.id]; 0 


-- receive count value from kernel 
linkin ?  short.header; data.size::data.array 


mm nm cr rs rt crm re te rr i ma a i a i a i a a SS aS See 


a ee ee ee et ee ee et es ss es a mee cm se ec mm me me st ee ee 


PROC advance ([2]CHAN OF ANY link, 
VAL INT count. 1) 


SS —— 2 a Ss Gee eee ent a eee ee a me es eee a ee eee Ce et ee ee ee ee ee ee eee se 


{{{ description 
-- This procedure requests that the distributed kemel 
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—- increment the specified event counter. 


cs SS ES CS TS SS A SS SS SN NS CSG) GSS Sc cc cc es et me me ey ee ee ee ee ee ee ee a a a ee eS Se Se 


{{{ labraries 
#USE "\ecslib\ecssymb.tsxr" 
es 


{{{ declarations 

— altemate name for channel 

CHAN OF ANY linkowut "ES Janie( 1) 
ay 


— request the advance action 
linkout ! ([(ADVANCE.CMD, count.id]; 0 


<9 a ee ee ep > > ee ee eee, ee oe ce Se ee ee ee ee 0 ee ee a a ee ee oe oe a en a een ee ee ee 


Se es cc cs ee ee ee em ee i es ee es ee ee ee ee ee ee eS ee ee ne ee ee ee ee 


PROC await ([2]CHAN OF ANY link, 
VAL INT Gount «dy 
VAL INT count . value) 
{{{ description 
—- Perform the await function on the specified event count and 
—-- await the argument count value. 


-- Note that the waiting process table (node.awaits) in the 

-- kernel that maintains the event count may be full. 

-—- In this case, the kermel retums a message to this process 

-—- identifying that this is the case. This procedure then 

-—- waits a period of time and retransmits the await request to 
-~ the kernel. If the await request is again rejected due to 

-~ a full waiting process table, the wait time is doubled before 
-—- retrying the await request. This process will continue until 
-- either the requested wait-for count is reached or the await 
—- request is accepted and placed in the kermel's waiting process 
=) table. 


— oe Se ee Pe ee ee es me es ee ey es es re es es ee ee ee ee es ee a a 8 ewe Ss eS 


{{{ libraries 
#USE "\ecslib\ecssymb.tsr" 
}}} 


{{{ declarations 

— altemate channel names 
CHAN OF ANY linkin ES link(0] 
CHAN OF ANY linkout 2S) eink [1] 


— commnications data structures 
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[ZINE short .header : 


INT return.tag IS short .header [0] 
VAL []BYTE data.array RETYPES count.value : 
INT cummy.size : 

[4] BYTE dummy.array : 


-—- structures for controlling retransmit of rejected awaits 
TIMER clock =: 

INT current .time, wait.time : 

bh} 


SEQ 
—- request await from kernel 
linkout ! [AWAIT.CMD, count.id]; (SIZE data.array) ::data.array 
-—- receive reply from the kernel 
linkin ? short.header; dummy.size: :dumny.array 
{{{ check to see if the await was accepted 
102 
return.tag = AWAIT.QUE.FULL 
{{{ wait by binary back-off and retry the await request 


SEQ 
== initial walt time (short) 
wait.tim :=1 
WHILE return.tag = AWAIT.QUE.FULL 
SEQ 
clock ? current .time 
clock ? AFTER (current .time PLUS wait .time) 
-~ set-up the doubled delay time 
wait .time := wait.time * 2 
—- request await from kernel 
linkout ! [AWAIT.CMD, count.id]; (SIZE data.array) ::data.array 
-—- receive response from kemel 
linkin ?  short.header; dummy.size: :dummy.array 
bh} 
ELSE 
{{{ do nothing -- await was accepted 
SKIP 


bh} 
b}} 


= ee ec em n> ee me > me me ee ee ee ee es ee em ee i a a ns cm ee ee 2 
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PROC ticket ([2]CHAN OF ANY link, 


VAL INT Count. 1d, 
INT count. value) 
{{{ description 


-- This procedure request a reservation or 'ticket' from the 
-- distributed kernel for the specified event count/sequencer. 
-- The value of the ticket is returned in the count.value 


eis 


{{{ libraries 
#USE "\ecslib\ecssymb.tsr" 
39; 


{{{ declarations 
— altermate channel names 


CHAN OF ANY linkin IS laink[0] 

CHAN OF ANY linkout roe aink (1) 

— commmication data structures 

i2Z) INT short .header ; 

INT data.size : 

[] BYTE data.array RETYPES count.value : 
Js 

SEQ 


-- request ticket from kermel 

iemkouc '§ ([TICKET.CMD, count.id]; 0 

— receive ticket from kernel 

linkin ?  short.header; data.size::data.array 


G PUT PROCEDURE 


PROC put ({2]CHAN OF ANY link, 


VAL INT Count 1G, 
VAL INT index, 
VAL []BYTE data.array) 


{{{ description 

— This procedure writes data to the shared memory segment 
-- associated with the argument event count. The data array 
-- is written offset from the start of the shared memory 

— segment by the number bytes specified by the index 

== parameter. 


—_ ee ee ee i 2 ee es 2 ee me er ee ee ee Se ee aS OS SS 


{{{ libraries 
#USE "\ecslib\ecssymb.tsr" 
b}} 


{{{ declarations 
— alternate channel nare 


CHAN OF ANY Linkout Is isbelie | (L] 


— convert the index value to an array of bytes for 


eo 


— transmission as data 
VAL (]BYTE index.array RETYPES index :; 


—- identify the size of the data arrays 


VAL INT index.size IS SIZE index.array : 
VAL INT data.size ES SIZE data.array : 
VAL INT local.size IS index.size + data.size : 


— define a local array for the data and index value 
[MAX .MESSAGE.SIZE + index.size] BYTE local.array : 
ea) 


SEQ 
—- combine the data array and the array representation of 
-- the index value into a single array 
[local.array FROM 0 FOR index.size] := index.array 
[local.array FROM index.size FOR data.size] := data.array 
-- request kermel to store the data array 
linkout ! (PUT.CMD, count.idj; local.size::local.array 


— 0 Se a ae ee a) a ee ee ee ee ce 2 ne ee om ee Se ee eee es em Se ae ee es ee es i a ee ee 


a ee ee es ee es ee et ee ee i a ee i es Se ee ee eee eee ee 


PROC get ((2]CHAN OF ANY link, 


VAL INT count .id, 
VAL INT index, 


[] BYTE data .array) 


{{{ description 

-- This procedure performs a read of the shared memory 

— segment associated with the argument event count. 

-~ The number of bytes read is determined by the size 

~~ of the argument data array. The bytes read from 

-- the shared memory are offset from the start of the 

-~ shared memory segment by the number of bytes specified 
-- in the index parameter. 

Jeet 








{{{ libraries 
#USE "\ecslib\ecssymb.tsr" 
b}} 


{{{ declarations 

— altemate name for the channels to kermel 
CHAN OF ANY linkin 16S) link [0] 
CHAN OF ANY lLinkout tS link [1] 


—— commmications data structures 
(2) INT short .header : 
INT data.size : 
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— define the size and index retrieval parameters as 

— an array of bytes for sending as data 

VAL []BYTE get.specs RETYPES [index, SIZE data.array] 
VAL INT spec.size is SIZE get.specs : 

oe 


SEQ 
-—- request the shared memory read operation 
linkout ! {[GET.CMD, count.id]; spec.size::get.specs 
—- receive the results of the read operation 
linkin ?  short.header; data.size::data.array 


Jie) JI 


APPENDIX D 
SAMPLE PROGRAM USING THE SHARED-MEMORY INTERFACE 


A DESCRIPTION 

This appendix provides an example of the methodology employed 
to write a program using the shared-memory interface developed for 
this thesis. The programming example selected to demonstrate the 
methodology is the bounded buffer problem. In this problem, a 
producer passes data to a consumer via a bounded buffer or queue. 
The buffer serves to “smooth out” variations in the data production 
and consumption rates. To add slightly to the single producer 
bounded buffer problem, this implementation of the problem will 


provide for two producers of data. 


B MODULARIZATION 

This particular problem can be naturally subdivided into three 
basic modules. Each of these are listed and described below. 

1. Producers 

Each instantiation of this module generates a continuous 

stream of data elements at a specified average rate with some 
characteristic random variation in the rate. The data elements 
produced are placed in a segment of memory shared with the 
consumer of the data. Access to the shared-memory segment is 
controlled using an event count and a sequencer. The sequencer 


controls the access of the two producers to the buffer. The associated 
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event count is advanced when either producer uses its ticket to add 
fata to the butier. 
2. Consumer 

This module continuously removes data elements from the 
shared memory buffer at a specified average rate with some 
characteristic random variation in the rate. The average rate for the 
single consumer should be at least equal to the total of the producer’s 
rates. If not and the consumer can not keep up with the producers, 
the buffer will eventually fill and the producers will be forced to 
remain idle while waiting for the consumer. 

An event count associated with the consumer is advanced 
when a data element is removed from the buffer. Note that if the 
producer and consumer event counts are initially equal, the number of 
data elements in the buffer will be equal to the difference between the 


event counts. 


C MODULE CODING 

The code for each of the program modules should then be 
developed. The following sections provide an abbreviated listing of the 
code for the modules. 


1. Producers 


PROC producer ([2]CHAN OF ANY link, 


VAL INT in, Out) 
#USE "\ecslib\ecsproc.tsr" -- the interface library 
#USE “globals.tsr" -- global constants 
INT my.ticket : 
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[data .element .size] BYTE data.element : 


WHILE TRUE 
SEQ 


—- insert code to produce a data element 


ticket (link, in, my.ticket) 
await (link, in, my.ticket) 
await (link, out, 
(my.ticket - data.buffer.size) + 1) 
put (link, in, 
(my.ticket + 1)\data.buffer.size, data.element) 
advance (link, out) 


SS ee ewe ee a a ee ee a ee a Se a ee 2 es ee eee 


2. Consumer 
PROC consumer ([2]CHAN OF ANY link, 
VAL INT 419 Out) 

#USE "\ecslib\ecsproc.tsr" -- the interface library 
#USE “globals.tsr" -- global constants 
INT my.count : 
[data .element .size] BYTE data.element : 
SEQ 

my.count := 0 

WHILE TRUE 

SEQ 


My COUN i= Mmy- counts! 
await (link, in, my.count) 
get (link, out, 
my.count\data.buffer.size, data.element) 
advance (link, out) 


-- insert code for consuming the data element 


D. APPORTIONMENT OF MODULES 
To best demonstrate the nature of the abstracted programming 
interface, the apportionment of modules will be done in two different 


ways. The first way will assign all modules, shared memory, and event 
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counts to a single Transputer. The second way will distribute the 


modules, shared memory, and event counts on three different 


Transputers. 


1. Single Transputer 


Library aport.tsr 


#USE "“globals.tsr" -- global constants 
— symbolic constants for counts 

VAL in 18S Or: 

VAL out is i tars 


— count node assignments (which node maintains the count) 


VAL count.node Is {[0, 0] 


— shared memory segment sizes 


VAL count.size is [0, data. buffer.size] 


— network adjacency matrix to match particular 
— physical configuration. This matrix matches 
— a clockwise ring on a B003 board. 





VAL node.link Is (1Ob2,2,21)7 
fp 072,27 
[2,2,0,2], 
(2,2,2,0]] 
PROC node0() -—- procedure to run on Transputer 0 


ea SS SS eS ae eS a Se ee ee ee ee ee eS SS eS = 


#USE "\ecslib\ecsproc.tsr"  —-—- 
#USE “procs.tsr" -- 
#USE "“aport.tsr" - 


VAL INT node.id 

[8] [2]CHAN OF ANY links : 

CHAN OF ANY pro.one. link 
CHAN OF ANY pro.two. link 
CHAN OF ANY consume. link 
PRI PAR 


ecs.kernel (links, node.id, 


the interface library 
library of procedures 
apportionment structures 


to. Om. 
IS links [4] 


toe lInks [5] 
iS lanks [6] 


count .node, count.size, 


node. link [node .id)) 


PAR 


producer (pro.one.link, in, out) 
producer (pro.two.link, in, out) 
consumer (consume.link, in, out) 
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2. 


Three Transputers 


Library aport.tsr 


DE > OS CD Oe > eS ee oe 2 ee ee ee ee ee ee ce ee Cs ee er ee ee ee ee ee eee eo 


#USE “globals.tsr" -- global constants 
—- symbolic constants for counts 

VAL in IS OF: 

VAL out TS A ae 


— count node assignments (which node maintains the count) 
VAL count .node IS (Ope 24 


— shared memory segment sizes 
VAL count.size ES (0, data. buffer.size] 


——- network adjacency matrix to match particular 
-- physical configuration. This matrix matches 
~——- a clockwise ring on a BO03 board. 
VAL node.link to” (T(07272721, 

(2,0,2,2], 

(2,2,0,2], 

(2,2,2,0]] 


— So eee So 0 ae Se ee a eee 


PROC nodeQ() -- procedure to run on Transputer 0 


es 69S eS a ee ee a ee SS SO ee 





#USE "\ecslib\ecsproc.tsr" -- the interface library 
#UGH: “PrEOScs tsi! -- library of procedures 
#USE “aport.tsr" -- apportionment structures 
VAL INT node .id NES) ONES 

[5] (2]CHAN OF ANY links : 

CHAN OF ANY pro.one.link IS links([4] 

PRI PAR 


ecs.kernel (links, node.id, 
count.node, count.size, 
node. link [node .id]) 

producer (pro.one.link, in, out) 


PROC nodel() -- procedure to run on Transputer 1 
#USE "\ecslib\ecsproc.tsr" -— the interface library 
#USE “procs.tsxr" -- library of procedures 
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USE abort cst" -- apportionment structures 


VAL INT node .id ac esl 

(5] [2] CHAN OF ANY links : 

CHAN OF ANY pro.two.link IS  links(4] 
PRI PAR 


ecs.kernel (links, node.id, 
count .node, count.size, 
node. link [node .id]) 
producer (pro.two.link, in, out) 


PROC node2() -- procedure to mim on Transputer 2 
#USE "\ecslib\ecsproc.tsr" -- the interface library 
#USE "procs.tsr" -- library of procedures 
#USE "aport.tsr" -—- apportionment structures 
VAL INT node .id ig: 
(5] [2] CHAN OF ANY links : 
CHAN OF ANY consume.link IS links [4] 
PRI PAR 


ecs.kernel (links, node.id, 
count .node, count.size, 
node. link [node .id]) 
consumer (consume.link, in, out) 


oe 


APPENDIX E 


DETAILED SOURCE CODE FOR THE MESSAGE-PASSING ABSTRACT 
INTERFACE LIBRARY 





A. SYMBOLIC CONSTANTS 


{it Li 

ice Sabrary > 

{{{ VAL declarations 

{{{ useful symbolic constants 


VAL ELSE tS) fee: 
VAL NIL CS ease es 
VAL SENDER 1S) UR 
VAL OUTPUT nS eee: 
VAL RECEIVER ESL 
VAL INPUT cS. oh 


{{{ system limits 
-— define the maximum communications data array size 
VAL MAX .MESSAGE .SIZE TS 10242 


-- define the maxinum number of system nodes 
VAL MAX. NODES iS a7Gee 


-- define the maxinum number of global channels 
VAL MAX .CHANS TS s2560-. 


-- define the mazimm number of channels per node 
VAL MAX.PROC.PER.NODE IS 40 : 


-~ define the mazimm number of processes per node 
VAL MAX.CHAN.PER.NODE IS 40 : 

ye 

{{{ packet tags used during initialization 


VAL INT Scan BBs elle 
VAL START .ACK PS ce 
VAL INIT.DATA IS 
VAL DATA. ACK Se 
VAL INET -ofOr cS: oe 
VAL oLOP ACK ES 236m 
VAL IND L.QUIT, Looe 


}}} 
{{{ packet tags used during canmmnication 
VAL RECEIVER. READY IS, ae 


VAL SEND [iS (053 
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b}} 
as 
ee 


KERNEL PROCEDURE 


PROC csp.kernel (VAL INT node .id, 
VAL [J] INT node. link, 
VAL [] (2] INT chan .map, 


(]CHAN OF ANY loc.chan) 
{{{ description 
-- This procedure is the distributed kernel for a message passing 
-- based model programming interface. This procedure should be 
-- executed in parallel with application program modules using the 
-- interface. This procedure should be run at high priority and 
-- the application modules run at low priority. 


{{{ libraries 
#USE “cspsymb.tsr" 
}}} 


{{{ declarations 
{{{ local abbreviations 


VAL INT no.h.links§ IS 4; 

VAL INT no.s.links IS SIZE chan.map : 

VAL INT no.1l.chan IS SIZE chan.map : 

VAL INT no. links ins no.h.links + no.s.links : 
VAL INT num.nodes IES: SIZE node. lank”; 


es 

{{{ channel declaration and placement 
(no.h.links*2]CHAN OF ANY hard.links: 
PLACE hard.links AT 0 : 


(MAX .CHAN.PER.NODE] (Z2]CHAN OF ANY links: 

b}} 

{{{ communications data structures 

-- definition of the communications header parts 


(3] INT header : 

INT header.action.code IS header (0] 

INT header .to.node tS header [1] 

INT header .gchan ins header [2] 

-- definition of the communications packet data segment 
INT data.size : 

(MAX .MESSAGE .SIZE] BYTE data.array : 


}}} 

{{{ channel mapping data structures 

-- kernel data structures for communications routing and 
ea Wahlagemnenie 

(MAX .CHANS] (2] INT gchan.node, 


log 


gchan.lchan : 
[MAX .CHANS ] INT lchan.gchan : 
Ip 
i 


{{{ procedures 
{{{ PROC get.local (local, return) 


mew em ee ee ee ee mm em Se oS ee ees Se ee ee ee ee a ee 


one GE 9 oe Ce © > ee es ee > ee rs ee oe ee es ee ns a: Se a oe ee 


{{{ description 

-- This procedure broadcasts local channel map data 
-—- to all nodes for network global structure 

—- initialization 


{{{ declarations 

-- for receiving acknowledge of remote broadcast 
INT acknowledge : 

b}} 


SEQ 
{{{ broadcast start signals 
— Open a path to all nodes in the system. 
— Intermediate nodes receiving these start 
— tokens and relaying them will not shut-down 
— until a corresponding stcp token is received. 
SEQ to.node = 0 FOR num.nodes 


IF 
to.node < node.id 
SEQ 
lecak !  [INIT.START, to.node, node.id]; 0 
return ? acknowledge 
ELSE 
SKIP 


a) 
{{{ broadcast all local link data 
— scan the channel map and build the local structures 
SEQ chan.no = 0 FOR no.l.chan 
{{{ local abbreviations 


VAL (JINT  global.ident RETYPES chan.map[chan.no] 

VAL (]BYTE ident.array RETYPES chan.map(chan.no] 

VAL INT global chan .1d vic glebal .ident [0] 

VAL INT chan .mode 1ES) global .ident (1] 

VAL INT link.no IDS chan .no temeeh. links : 
bh} 

SEQ 


—— build local data structures 
gchan .node [glebal.chan.id] [chan.mode] := node.id 
gchan.1chan (global .chan.id] (chan.mode] := link.no 
lchan.gchan(link.no] := global.chan.id 
-- broadcast to other nodes 
SEQ to.node = 0 FOR num.nodes 

IF 
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to.node <> node.id 


SEQ 
local ! [(INIT.DATA, to.node, node.id]; 8::ident array 
return ? acknowledge 
ELSE 
SKIP 


eh 

{{{ broadeast done signals 

— notify other nodes that done with init transmission 
SEQ to.node = 0 FOR num.nodes 


a> 
to.node <> node.id 
SEQ 
local ! [INIT.STOP, to.node, node.id); 0 
return ? acknowledge 
ELSE 
SKIP 


Pr} 

{{{ send local quit signal 

— notify that local transmissions over 
local ! [INIT.QUIT, node.id,node.idj; 0 


ie! 


}}} 
{{{ PROC get.remote (remote, return) 


PROC get.remote (CHAN OF ANY remote, return) 

{{{ description 

-- This procedure inputs remote nodes' channel map data 
-- to all nodes for network global structure 

-—- initialization 


{{{ local declarations 
(3) INT header : 


INT init .code ILS header [0] 
INT to.node.id TS header [1] 
INT from.node.id IRS header [2] 
INT data.size : 

ZN global.ident : 

(] BYTE ident .array RETYPES global.ident : 
INT Gliobalvechans.1d 1S global . ident [0] 
INT chan .mode iS global .ident [1] 
INT Gustscount > 

lees 

SEQ 


{{{ initialize 
— will receive at least 2 messages from every other node 
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quit.count := 2*(num.nodes - 1) 
re 
WHILE quit.count > 0 
ALT link.no = 0 FOR no.h.links 
-- get a message from somewhere 
links [link.no] [INPUT] ? header; data.size::ident .array 
le 
{{{ for my node, process it 
to.node.id = node.id 
SEQ 
to.node.id := from.node.id 
-—- categorize and act on message 
183; 
{{{ start message 
init.code = INIT.START 
SEQ 
-- acknowledge message to remote node 
init.code := START.ACK 
remote ! header; data.size::ident.array 
}}} 
{{{ (Stareese. 
init.code = START.ACK 
-- let local process know the ack rec'd 
return ! NIL 
bh} 
{{{ data message 
init.code = INIT.DATA 
SEQ 
gchan .node [global .chan.id] [chan.mode] := from.node.id 
-~ acknowledge message to remote noce 
init.code := DATA.ACK 
remote ! header; data.size::ident.array 
}}} 
{{| cakavack 
init .code = DATA.ACK 
-- let local process know the ack rec'd 
return ! NIL 
}}} 
{{{ stop message 
init.code = INIT.STOP 
SEQ 
-- one less message to get fram a remote node 
quit.count := quit .count==— 
-- acknowledge message to remote node 
init .code := STOP.ACK 
remote ! header; data.size::ident.array 
}} 
{{{ stop ack 
init.code = STOP.ACK 
SEQ 
-- one less message to get from a remote node 
quit. CoOune 2 = Glltt ,count ea 
-- let local process know the ack rec'd 
return ! NIL 
bh} 
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Dae 
{{{ for a different node, forward it 


ELSE 
SEQ 
remote ! header; data.size::ident.array 
103 
init.code = INIT.START 
— reserve a path for the sender 
Qui couness= Guit.count + 1 
init .code = START.ACK 
SKIP 
init.code = INIT.DATA 
SKIP 
init.code = DATA.ACK 
SKIP 
init .code 
SKIP 
init.code = STOP.ACK 
-- done with pass-thru reservation 
Gui COUnE = oat count =— 1 
ELSE 
SKIP 


GGT sober 


here 
{{{ send local quit signal 
— done with remote receives 
remote ! [INIT.QUIT, node.id, node.idj; 0 
}}} 


2 eC cy ee ee ee ee ee ee ee ee ee ee es ee ee ee ee ee ee SS a ee ee ee ee 


{{{ description 

-- This procedure multiplexes locally generated initialization 
-—- messages and 'pass-through' initilization messages onto 

— the hardware comminication links 


{{{ local declarations 
—— packet header definition 


(2 INT. header : 
INT init .code 1s header [0] 
INT to .node.id IS header [1] 


-- init packet data 


INT data.size : 
(2] INT global.ident : 
(] BYTE ident .array RETYPES global.ident : 


=alocal flags 
BOOL local .done, 
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remote.done : 
Je) 


SEQ 
{{{ initialize 
local.done := FALSE 
remote.done := FALSE 
}}} 
WHILE (NOT local.done) OR (NOT remote.done) 
PRI ALT 
{{{ remotely generated messages 
(NOT remote.done) & remote ? header; data.size::ident .array 
IF 
init .code = INIT.QUIT 
remote.done := TRUE 
ELSE 
links [node.1link[to.node.id]] [OUTPUT] ! header; 
data.size::ident.array 
ea) 
{{{ locally generated messages 
(NOT local.done) & local ? header; data.size::ident .array 
IF 
init.code = INIT.QUIT 
local.done := TRUE 
ELSE 
links [node. link [to.node.id]] [OUTPUT] ! header; 
data.size::ident.array 
Pe 


ES SS SD ce et ee ee es ee ee ee ee ee ee Se) ee ee ee mm ew et er a ome 


i 
{{{ PROC map.out (local.link, local.chan) 





mee a es ee a ee Se ee a a ee 


PROC map.out((2]CHAN OF ANY local.link, 

CHAN OF ANY local .chan) 
{{{ description 
-- Perform the data transfers needed to connect the output 
-- end of a global channel to the receiving local channel 
38) 


{{{ declarations 

-- alternate name for the local bidirectional channel 
CHAN OF ANY link.in IS local.link [0] 

CHAN OF ANY link seut TS “wigeal slink (a) 


— for receiving signal to stare transmissicy 
INT any : 


-- data structure for transmitted message 
INT data.size : 
[MAX .MESSAGE . SIZE] BYTE data.array : 
py 
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WHILE TRUE 
SEQ 

PAR 
-— know that receiver is ready to receive 
link.in ? any 
-—- get the transmitted data 
local.chan 7? data.size::data.array 

-—- send the data via the kernel 

link.out ! SEND; data.size::data.array 


Pt 
{{{ PROC map.in (local.link, local .chan) 
PROC map.in([{2]CHAN OF ANY local. link, 
CHAN OF ANY local .chan) 
{{{ description 
-- Perform the data transfers needed to connect the input 
—- end of a global channel to the transmitting local channel 


{{{ declarations 

— alternate names for the local channels 
CHAN OF ANY lagi epee! Toe locale lank (G] 
CHAN OF ANY link .out 1S local link(1) 


—- data structure for received data 


INT data.size : 
[MAX .MESSAGE.. SIZE] BYTE data.array : 
have 
WHILE TRUE 

SEQ 


-—- identify to the kermel that the process is ready to receive 
link.out ! RECEIVER.READY; 0 

-- receive the transmitted data fram the kernel 

link.in ? data.size::data.array 

—- send data to the application 

local.chan ! data.size::data.array 


{{{ PROC buffer.1link (chan.in, chan.out) 


ee ce a em me we Se me sm me es me STS 


PROC buffer.link (CHAN OF ANY in.link, out.link) 


{{{ description 

—- This procedure provides a buffer for hard link input. 

-- This buffer increases the overall throughput of the node. 
ae 
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{{{ declarations 


[MAX .MESSAGE .SIZE] BYTE data.array.0, 
data.array.1 : 
eine ye header. 0, 
header.1 : 
INT data.size.0, 
data.size.1 : 
}}} 
SEQ 
in.link ? header.0; data.size.0::data.array.0 
WHILE TRUE 
SEQ 
PAR 


out.link ! header.0; data.size.0::data.array.0 

in.link ? header.1; data.size.1::data.array.1 
PAR 

out.link ! header.1; data.size.1::data.array.1 

in.link ? header.0; data.size.0::data.array.0 


— Se ew SSS SS a a em ee me a ee ee es ee ee es ee ee es ee 2 


eee ec ec ee se ee i ee ee ee ee ee ce se ee ee ee ce ee es ae ee ee ss ee 


PROC build.packet (VAL INT link.no) 

{{{ description 

-— Based on the local channel sending the message to the kernel, 
-—=- construct an intemode cammmications packet. 


a a ee a a a ce ee a Se eee a a ae a eee ae a ee ee ee ee ss ee SS 66 6 eC ee Se a EP ee ee a oe Se 


SEQ 
— identify the global channel being used 
header .gchan := lchan.gchan (link .no] 
— find the node to which the packet is to be routed 
IF 
header .action.code 
header .to.node gchan .node [header .gchan] [OUTPUT] 
header .action.code = SEND 
header.to.node := gchan.node[header.gchan] [INPUT] 
ELSE 
header.to.node := node.id 


RECEIVER. READY 


mm mm me cr ee a se ee ce we ee es ew ww ee ne 


{{{ description 
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-- This procedure perfomns communications routing and maintains 
-- synchronization between the sending and receiving processes 
-- by sequencing commmnications between the local and remote 

-- processes. 


{{{ packet for this node 
header.to.node = node.id 
103 
header.action.code = RECEIVER.READY 
— let the sender know that receiver is ready 
links [gchan.1chan (header.gchan] [CUTPUT]] [OUTPUT] ! SEND 
header .action.code = SEND 
— pass on the data packet to its destination 
links [gchan.lchan [header.gchan] [INPUT] ] [OUTPUT] ! 
data.size::data.array 
Pe 
{{{ packet for other node - forward it 
ELSE 
links (node. link [header .to.node]] [OUTPUT] ! header; data.size::data.array 


net 


PAR 
{{{ buffer hardware links 
PAR link.no = 0 FOR no.h.links 
PAR 
buffer. link (hard. links [link.no + no.h.links], links [(link.no] [INPUT] ) 
buffer . link (links [link.no] [OUTPUT], hard.links [link.no]) 
s}} 
SEQ 
{{{ initialization 
{{{ initialize kernel data structures 
SEQ i = 0 FOR MAX.CHANS 


SEQ 
lchan.gchan[i] := NIL 
gchan .node [1] = (NIL, NIL] 
gchan.lchan[i] := [(NIL,NIL] 


}}} 
{{{ query local channels to build data structures 
{{{ declare local channels for initialization 
CHAN OF ANY Lecal, 
Lenore, 
rerun |: 
ae 
PAR 
get . local (Local, return) 
get.remote (remote, return) 
multiplex(local, remote) 
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yh} 
b}} 
PAR 
{{{ build mapping processes 
-- Create the channel controller processes to connect the 
-- local channels to global channel ends 
{{{ local declarations 


TIMER clocks 
INT ture: 
}}} 

SEQ 


{{{ wait for init messages clear the network 

-~- This is a cobble to correct a problem with one 

-——- node completinig initialization and sending a 'real' 

-- message that is not properly handled by a node 

-- that has not yet been initialized. Just wait for 

-- a long time for the initilization messages to 

-—~ clear the network 

clock ? time 

clock ? AFTER time + 5000000 -- wait 5 seconds 

Py 

PAR chan.no = 0 FOR MAX.CHAN.PER.NODE 

IF -= this channel is among those declared 
Chamene.— re cham 
IF -- input or output global channel end 
chan .map[(chan.no] [1] = INPUT 
map.in (links [chan.notno.h.links], loc.chan [chan .no]) 
ELSE 
map .out (links (chan .notno.h.links],loc.chan{chan.no]) 
ELSE 
SKIP 

a 
{{{ monitor communications 
— accept input from channel controllers or from hard links 
{{{ local declarations 
-- The communications monitoring procedure uses a 'fair' implementation 
-- of the ALT structure. In general, it provides that if a communication 
-- was just received from one of several channels, that channel will have 
—~ the lowest priority for the next execution of the ALT. In this way, 
-- no single communications channel cam 'starve' access to the kernel 
-- from the other commmications channels. The variables defined 
-- below are used to track the last commmicating channel for this 
-- 'fair' ALT. Note that hard and soft links are treated separately. 


my last, last.s: 
ie) 
SEQ 
{{{  initializacicn 
last.s := 0 
last.h := 0 
le) 
WHILE TRUE 
PRI ALT 
{{{ handle external hardware links 
ALT link.no = 0 FOR no.h.links 
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links [(Link.no + last.h)\no.h.links] [INPUT] ? header; 
data.size: :data.array 
SEO 
process.packet ((link.no + last.h)\no.h. links) 
last.h := (no.h.links - 1) + link.no 
De! 
{{{ handle local soft links 
ALT i = 0 FOR no.s.links 
lankstar last.s) mo.s.links) + no.h.links) [INPUT] 2 
header .action.code; 
data.size: :data.array 
VAI IND slamkene Io ((i+ last.s) \no.s.links) + no.h.links : 
SEQ 
build.packet (link .no) 
process .packet (link .no) 
last.s := i+ (no.s.links - 1) 
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APPIENDEX F 
SAMPLE PROGRAM USING THE MESSAGE-PASSING INTERFACE 


A. DESCRIPTION 

This appendix provides an example of the methodology employed 
to write a program using the message-passing interface developed for 
this thesis. The programming example selected is the same bounded 
buffer problem used in Appendix D to demonstrate programming with 
the shared-memory interface. This problem consisted of two produc- 


ers and one consumer with a bounded buffer or queue between them. 


B MODULARIZATION 

The modularization of the bounded buffer problem for program- 
ming under the message-passing interface is similar to that specified 
for programming with the shared memory interface. As with the 
shared memory modularization, the producers and consumer are indi- 
vidual modules. However, with the shared memory interface, the 
buffer between the producers and consumer is a natural consequence 
of the memory shared between the modules. In the message-passing 
scheme, however, this buffer must be defined and coded as a separate 
module. 

i: Exe er 

Each instantiation of this module generates a continuous 


stream of data elements at a specified average rate with some charac- 


Lae 


teristic random variation in the rate. The data elements produced are 
output to a buffering process via a global communications channel. 
2. Consumer 
This module continuously attempts to input data elements 
from a global communications channel. Data elements input are con- 
sumed at a specified average rate with some characteristic random 
variation in the rate. The average rate for the single consumer should 
be at least equal to the total of the producer’s rates. If not and the 
consumer can not keep up with the producers, the buffer will eventu- 
ally fill and the producers will be forced to remain idle while waiting 
for the consumer. 
3. Buffer 
This module inputs communications from two global channels 
and queues the input data elements for output on a third global com- 
munications channel. Internally, this buffer should exhibit first-in, 


first-out queue characteristics. 


C MODULE CODING 

The code for each of the program modules should then be devel- 
oped. The following sections provide an abbreviated listing of the 
code for the modules. 


1. Producers 


PROC producer (CHAN OF ANY output) 


#USE "\csplib\cspproc.tsr" -- the interface library 
#USE “globals.tsr"™ —- global constants 


lol 


[data.element .size] BYTE data.element : 


WHILE TRUE 
SEQ 


-- insert code for producing a data element 


output ! data.element 


a ee ce ee ee a Se a a Se a ce eS Se eee ee 


PROC consumer (CHAN OF ANY input) 


cy mm es ee i ee ee ss ee ee ee ee a ee a es ee 


#USE "\csplib\cspproc.tsr" -- the interface library 
#USE “globals.tsr" -~ global constants 
[data element .size] BYTE data.element : 
WHILE: TRUE 

SEQ 


input ? data.element 


-—- insert code to consume data element 


ee ee ee ee ee Se ee ee ee ee ee ee ee ee es ae ae 


3. Buffer 


PROC buffer (CHAN OF ANY buff.in.1, buff.in.2, buff.out) 








— Note, procedure written in a manner to illustrate sare 
— features of the OCCAM programming language; not for 


— efficiency 
#USE "\csplib\cspproc.tsr" -- the interface library 
#USE “globals.tsr" ~- global constants 
[buffer.size]CHAN OF ANY butt (cham. 
PAR 


-- accept input from either producer 
[data.element.size]BYTE data.element : 


WHILE TRUE 
ALT 
buff.in.1 ? data.element 
buff.chan{0}] ! data.element 


Lia 


buff.in.2 ? data.element 
buff.chan[0] ! data.element 


-- buffer the input 
PAR 1 = 0 FOR buffer.size - 1 
(data.element.size]BYTE data.element : 


WHILE TRUE 
SEQ 
buff.chan[i] 7? data.element 
buff.chan[i +1] ! data.element 


-- send from the buffer 
(data.element.size]BYTE data.element : 


WHILE TRUE 
SEQ 
buff.chan[buffer.size - 1) ? data.element 
buff.out ! data.element 


APPORTIONMENT OF MODULES 


As is done for the shared-memory interface, the apportionment of 


modules for the message-passing interface is done in two different 


ways. The first way will assign all modules to a single Transputer. The 


second way will distribute the modules on four different Transputers. 


1. Single Transputer 


Library aport.tsr 


PD ee PE ee Se ee ee ee es se ee ee eee ee ee ee ee ee 


#USE “globals.tsr" -- global constants 


— symbolic constants for global channels 
VAL prol.global Se Oo: 
VAL pro2.global ES te 
VAL con.global ils 22 Ge 


— network adjacency matrix to match particular 
— physical configuration. This matrix matches 
— a clockwise ring on a B003 board. 


VAL node.link iS (O72, 2,21, 
(2; 072,72) f 
(2,2,0,2], 
[2,2,2,0]] 
PROC nodeQ() -- procedure to run on Transputer 0 
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#USE "\csplib\cspoproc.tsr"  —- 
#USE "procs.tsr" == 
#USE "“aport.tsr" ae 


VAL INT node .id 
VAL [] [2] INT chan.map 


the interface library 
library of procedures 
apportionment structures 


LSey Oa 

[ [prol.global, SENDER], 
[prol.global, RECEIVER], 
[pro2 .global, SENDER], 
[pro2 .global, RECEIVER], 
[con .global, SENDER], 
[con .global, RECEIVER] ] 


ts 


[SIZE chan.map]CHAN OF ANY loc.chan : 
-- abbreviations for convenience only 


CHAN OF ANY pro. one .ouE 
CHAN OF ANY buff .in.one 
CHAN OF ANY pro .two.out 
CHAN OF ANY buff .in.two 
CHAN OF ANY Butt Our 
CHAN OF ANY Conia 

PRI PAR 


IS 
Is 
is 
iS 
Is 
IS 


loc 
ikere: 
loc 
Ise 
ike-e3 
lee 


-chan [0] 
.chan [1] 
.chan [2] 
.chan [3] 
chan [4] 
.chan [5] 


csp.kernel (node.id, node.link[node.id], 


chan.map, loc.chan) 


PAR 
producer (pro .one.out) 
producer (pro .two .out) 


buffer (buff.in.one, buff.in.two, buff.out) 


consumer (con. in) 


2. Four Transputers 


Ow a 9 ee a eG ae ae a 0 a ae ae a as ee ie a ee a a Se Se eee oe ee 


Library aport.tsr 


#USE “globals.tsr" -- 


—- symbolic constants for global channels 


VAL prol.global tS: > 0ns 
VAL pro2.global TS pele: 
VAL con.global ISs 2: 


global constants 


— network adjacency matrix to match particular 
— physical configuration. This matrix matches 
— a clockwise ring on a B003 board. 
VAL node.link 1S. (((0,;2-2,210 
(2,0,2,2], 
(2,2,0,2], 
[2,2,2,0]] 


em me i ee a a ee ee ee ee a ee ae ae eee a ee 
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mm amg SS mm em mmm ec a i a 


PROC node0() -- procedure to run on Transputer 0 


#USE "\csplib\cspproc.tsr" -- the interface library 
#USE "procs.tsr" =—— library of procedures 
#USE “aport.tsr" —- apportionment structures 
VAL INT node. id eS Oe 

VAL {](2]INT chan.map IS [[prol.global, SENDER] ] 


[SIZE chan.map]CHAN OF ANY loc.chan : 
—~- abbreviation for convenience only 
CHAN OF ANY pro.one.out IS loc.chan(0] 


PRI PAR 
csp.kernel (node.id, node.link [node.id], 
chan.map, loc.chan) 
producer (pro.one .out) 


a ee ee nes ee ee ee ee es ee ee se ee ee ee ee ee ee ee 


PROC nodel() — procedure to run on Transputer 1 
#USE "\csplib\cspproc.tsr" — the interface library 
FUSS eOECCS-ESL -- library of procedures 
#USE “aport.tsr" —- apportionment structures 
VAL INT node .id PS eeele: 
VAL [] (2) INT chan .map IS [[pro2.global, SENDER] ]: 


[SIZE chan.map]CHAN OF ANY loc.chan : 
—- abbreviations for convenience only 
CHAN OF ANY pro.two.out IS loc.chan([0] 


PRI PAR 
csp.kernel (node.id, node.link[node.id], 
chan.map, loc.chan) 
producer (pro .two.out) 


Se ae ee © a a ee eee ee a a ee ee ee ee ee eee ee 


PROC node2() -- procedure to run on Transputer 2 
#USE "\csplib\cspproc.tsr" -—- the interface library 
#USE "procs.tsr" -—- library of procedures 
#USE “aport.tsr" —- apportionment structures 
VAL INT node.id TSer Zs 
VAL [] [2] INT chan .map IS [{prol.global, RECEIVER], 


[pro2.global, RECEIVER], 


NS 


[con .global, SENDER] ] 


{SIZE chan.map]CHAN OF ANY loc.chan :; 
-- abbreviations for convenience only 


CHAN OF ANY buff.in.one IS loc.chan(0] 
CHAN OF ANY butt ine cws IS” -loc-cnan i) 
CHAN OF ANY but our TS 166.¢han (2) 
PRI PAR 


csp. kernel (node.id, node.link [node.id], 
chan.map, loc.chan) 
buffer (buff.in.one, buff.in.two, buff.out) 





PROC node3() -- procedure to run on Transputer 3 
#USE "\csplib\cspproc.tsr™" -- the interface library 
#USE “procs.tsxr" -- library of procedures 
#USE “aport.tsr" -- apportionment structures 
VAL INT node. id TS mers a: 
VAL {] [2] INT chan.map IS {{con.glebal, RECEIVER)] 


[SIZE chan.map]CHAN OF ANY loc.chan : 
-- abbreviations for convenience only 
CHAN OF ANY Cone am IS loc.chan[Q] 


PRI PAR 
csp.kermel (node.id, node.link[{node.id], 
chan.map, loc.chan) 


consumer (con. in) 


a a es mm es ow a ee 2 a ee 2 ee 2 Se ee ee a ee 
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